Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space

IEEE Transactions on Affective Computing

Volumn 2, Issue 2, 2011, Pages 92-105

  Nicolaou, Mihalis A ^a   Gunes, Hatice ^a   Pantic, Maja ^a  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

continuous affect prediction; Dimensional affect recognition; emotional acoustic signals; facial expressions; multicue and multimodal fusion; output associative fusion.; shoulder gestures; valence and arousal dimensions

Indexed keywords

AFFECT RECOGNITION; EMOTIONAL ACOUSTIC SIGNALS; FACIAL EXPRESSIONS; MULTI-MODAL FUSION; OUTPUT-ASSOCIATIVE FUSION.; VALENCE AND AROUSAL DIMENSIONS;

FORECASTING;

FACE RECOGNITION;

EID: 80054842318     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/T-AFFC.2011.9     Document Type: Article

References (67)

1. N. Alvarado, "Arousal and Valence in the Direct Scaling of Emotional Response to Film Clips," Motivation and Emotion, vol. 21, pp. 323-348, 1997.
2. N. Ambady and R. Rosenthal, "Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: A Meta-Analysis," Psychological Bull., vol. 11, no. 2, pp. 256-274, 1992.
3. P. Baldi, S. Brunak, P. Frasconi, G. Pollastri, and G. Soda, "Exploiting the Past and the Future in Protein Secondary Structure Prediction," Bioinformatics, vol. 15, pp. 937-946, 1999. (Pubitemid 30103164)
4. S. Baron-Cohen and T.H.E. Tead, Mind Reading: The Interactive Guide to Emotion. Jessica Kingsley Publishers Ltd., 2003.
5. S. Bermejo and J. Cabestany, "Oriented Principal Component Analysis for Large Margin Classifiers," Neural Networks, vol. 14, no. 10, pp. 1447-1461, 2001. (Pubitemid 33101880)
6. L. Bo and C. Sminchisescu, "Twin Gaussian Processes for Structured Prediction," Int'l J. Computer Vision, vol. 87, pp. 28-52, 2010.
7. L. Bo and C. Sminchisescu, "Structured Output-Associative Regression," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2403-2410, 2009.
8. R.A. Calvo and S. DMello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications," IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 18-37, Jan.-June 2010.
9. G. Caridakis, K. Karpouzis, and S. Kollias, "User and Context Adaptive Neural Networks for Emotion Recognition," Neurocomputing, vol. 71, nos. 13-15, pp. 2553-2562, 2008.
10. G. Caridakis, L. Malatesta, L. Kessous, N. Amir, A. Raouzaiou, and K. Karpouzis, "Modeling Naturalistic Affective States via Facial and Vocal Expressions Recognition," Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 146-154, 2006. (Pubitemid 47113445)
11. G. Chanel, K. Ansari-Asl, and T. Pun, "Valence-Arousal Evaluation Using Physiological Signals in an Emotion Recall Paradigm," Proc. IEEE Int'l Conf. Systems, Man and Cybernetics, pp. 2662-2667, 2007. (Pubitemid 351409969)
12. G. Chanel, J. Kronegg, D. Grandjean, and T. Pun, "Emotion Assessment: Arousal Evaluation Using EEG's and Peripheral Physiological Signals," Proc. Multimedia Content Representation, Classification and Security, pp. 530-537, 2006. (Pubitemid 44743885)
13. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schroder, "Feeltrace: An Instrument for Recording Perceived Emotion in Real Time," Proc. ISCA Workshop Speech and Emotion, pp. 19-24, 2000.
14. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J.G. Taylor, "Emotion Recognition in Human-Computer Interaction," IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32-80, Jan. 2001. (Pubitemid 32287669)
15. R. Cowie, H. Gunes, G. McKeown, L. Vaclau-Schneider, J. Armstrong, and E. Douglas-Cowie, "The Emotional and Communicative Significance of Head Nods and Shakes in a Naturalistic Database," Proc. LREC Int'l Workshop Emotion, pp. 42-46, 2010.
16. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge Univ. Press, Mar. 2000.
17. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, L. Lowry, M. McRorie, L. Jean-Claude Martin, J.-C. Devillers, A. Abrilian, S. Batliner, A. Noam, and K. Karpouzis, "The Humaine Database: Addressing the Needs of the Affective Computing Community," Proc. Second Int'l Conf. Affective Computing and Intelligent Interaction, pp. 488-500, 2007.
18. H. Drucker, C.J.C. Burges, L. Kaufman, A.J. Smola, and V. Vapnik, "Support Vector Regression Machines," Advances in Neural Information Processing Systems, pp. 155-161, MIT Press, 1996.
19. P. Ekman, Emotions in the Human Faces, second ed. Cambridge Univ. Press, 1982.
20. P. Ekman and W. Friesen, "Head and Body Cues in Gyrus and Inferior Medial Prefrontal Cortex in Social Perception," Perceptual & Motor Skills, vol. 24, pp. 711-724, 1967.
21. K. Forbes-Riley and D. Litman, "Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources," Proc. Human Language Technology Conf. North Am. Chapter of the Assoc. Computational Linguistics, pp. 201-208, 2004.
22. N. Fragopanagos and J.G. Taylor, "Emotion Recognition in Human-Computer Interaction," Neural Networks, vol. 18, no. 4, pp. 389-405, 2005. (Pubitemid 40922647)
23. D. Glowinski, A. Camurri, G. Volpe, N. Dael, and K. Scherer, "Technique for Automatic Emotion Recognition by Body Gesture Analysis," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition Workshops, pp. 1-6, 2008.
24. D. Grandjean, D. Sander, and K.R. Scherer, "Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization," Consciousness and Cognition, vol. 17, pp. 484-495, 2008.
25. A. Graves and J. Schmidhuber, "Framewise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures," Neural Networks, vol. 18, pp. 602-610, 2005. (Pubitemid 43186580)
26. M. Grimm and K. Kroschel, "Emotion Estimation in Speech Using a 3D Emotion Space Concept," Proc. IEEE Automatic Speech Recognition and Understanding Workshop, pp. 381-385, 2005.
27. H. Gunes and M. Pantic, "Automatic, Dimensional and Continuous Emotion Recognition," Int. J. Synthetic Emotions, vol. 1, no. 1, pp. 68-99, 2010.
28. H. Gunes and M. Pantic, "Dimensional Emotion Prediction from Spontaneous Head Gestures for Interaction with Sensitive Artificial Listeners," Proc. Int'l Conf. Intelligent Virtual Agents, pp. 371-377, 2010.
29. H. Gunes, M. Piccardi, and M. Pantic, "From the Lab to the Real World: Affect Recognition Using Multiple Cues and Modalities" Affective Computing: Focus on Emotion Expression, Synthesis, and Recognition, pp. 185-218, I-Tech Education and Publishing, 2008.
30. S. Hochreiter, "Untersuchungen zu Dynamischen Neuronalen Netzen," diploma thesis, Institut Für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München, 1991.
31. S. Hochreiter, "The Vanishing Gradient Problem during Learning Recurrent Neural Nets and Problem Solutions," Int'l J. Uncertainty, Fuzziness and Knowledge-Based Systems, vol. 6, no. 2, pp. 107-116, 1998. (Pubitemid 128539033)
32. S. Ioannou, A. Raouzaiou, V. Tzouvaras, T. Mailis, K. Karpouzis, and S. Kollias, "Emotion Recognition through Facial Expression Analysis Based on a Neurofuzzy Method," J. Neural Networks, vol. 18, no. 4, pp. 423-435, 2005. (Pubitemid 40922649)
33. D. Jurafsky and J.H. Martin, Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition, second ed. Prentice Hall, 2008.
34. I. Kanluan, M. Grimm, and K. Kroschel, "Audio-Visual Emotion Recognition Using an Emotion Recognition Space Concept," Proc. 16th European Signal Processing Conf., 2008.
35. K. Karpouzis, G. Caridakis, L. Kessous, N. Amir, A. Raouzaiou, L. Malatesta, and S. Kollias, "Modeling Naturalistic Affective States via Facial, Vocal and Bodily Expressions Recognition," Lecture Notes in Artificial Intelligence, vol. 4451, pp. 92-116, 2007.
36. J. Kim, "Bimodal Emotion Recognition Using Speech and Physiological Changes," Robust Speech Recognition and Understanding, pp. 265-280, I-Tech Education and Publishing, 2007.
37. A. Kleinsmith and N. Bianchi-Berthouze, "Recognizing Affective Dimensions from Body Posture," Proc. Int'l Conf. Affective Computing and Intelligent Interaction, pp. 48-58, 2007.
38. D. Kulic and E.A. Croft, "Affective State Estimation for Human-Robot Interaction," IEEE Trans. Robotics, vol. 23, no. 5, pp. 991-1000, Oct. 2007. (Pubitemid 47577975)
39. R. Lane and L. Nadel, Cognitive Neuroscience of Emotion. Oxford Univ. Press, 2000.
40. R. Levenson, "Emotion and the Autonomic Nervous System: A Prospectus for Research on Autonomic Specificity," Social Psychophysiology and Emotion: Theory and Clinical Applications, pp. 17-42, John Wiley & Sons, 1988.
41. P.A. Lewis, H.D. Critchley, P. Rotshtein, and R.J. Dolan, "Neural Correlates of Processing Valence and Arousal in Affective Words," Cerebral Cortex, vol. 17, no. 3, pp. 742-748, 2007.
42. M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, "Abandoning Emotion Classes-Towards Continuous Emotion Recognition with Modelling of Long-Range Dependencies," Proc. Ninth Interspeech Conf., pp. 597-600, 2008.
43. G. McKeown, M. Valstar, R. Cowie, and M. Pantic, "The Semaine Corpus of Emotionally Coloured Character Interactions," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 1079-1084, 2010.
44. A. Mehrabian and J. Russell, An Approach to Environmental Psychology. MIT Press, 1974.
45. M. Nicolaou, H. Gunes, and M. Pantic, "Audio-Visual Classification and Fusion of Spontaneous Affective Data in Likelihood Space," Proc. IEEE Int'l Conf. Pattern Recognition, pp. 3695-3699, 2010.
46. M. Nicolaou, H. Gunes, and M. Pantic, "Automatic Segmentation of Spontaneous Data Using Dimensional Labels from Multiple Coders," Proc. LREC Int'l Workshop Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, pp. 43-48, 2010.
47. M. Nicolaou, H. Gunes, and M. Pantic, "Output-Associative RVM Regression for Dimensional and Continuous Emotion Prediction," Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, 2011.
48. A.M. Oliveira, M.P. Teixeira, I.B. Fonseca, and M. Oliveira, "Joint Model-Parameter Validation of Self-Estimates of Valence and Arousal: Probing a Differential-Weighting Model of Affective Intensity," Proc. 22nd Ann. Meeting Int'l Soc. for Psychophysics, pp. 245-250, 2006.
49. M. Pantic and L. Rothkrantz, "Toward an Affect Sensitive Multimodal Human-Computer Interaction," Proc. IEEE, vol. 91, no. 9, pp. 1370-1390, Sept. 2003. (Pubitemid 40890819)
50. I. Patras and M. Pantic, "Particle Filtering with Factorized Likelihoods for Tracking Facial Features," Proc. Int'l Conf. Automatic Face and Gesture Recognition, pp. 97-104, 2004.
51. B. Paul, "Accurate Short-Term Analysis of the Fundamental Frequency and the Harmonics-to-Noise Ratio of a Sampled Sound," Proc. Inst. of Phonetic Sciences, pp. 97-110, 1993.
52. S. Petridis, H. Gunes, S. Kaltwang, and M. Pantic, "Static versus Dynamic Modeling of Human Nonverbal Behavior from Multiple Cues and Modalities," Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 23-30, 2009.
53. M.K. Pitt and N. Shephard, "Filtering via Simulation: Auxiliary Particle Filters," J. Am. Statistical Assoc., vol. 94, no. 446, pp. 590-616, 1999.
54. J.A. Russell, "A Circumplex Model of Affect," J. Personality and Social Psychology, vol. 39, pp. 1161-1178, 1980.
55. K. Scherer, A. Schorr, and T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research. Oxford Univ. Press, 2001.
56. K. Scherer, "Psychological Models of Emotion," The Neuropsychology of Emotion, pp. 137-162, Oxford Univ. Press, 2000.
57. B. Schölkopf and A.J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. The MIT Press, 2001.
58. M. Schuster and K.K. Paliwal, "Bidirectional Recurrent Neural Networks," IEEE Trans. Signal Processing, vol. 45, no. 11, pp. 2673-2681, Nov. 1997. (Pubitemid 127766336)
59. K.P. Truong, D.A. Leeuwen van, M.A. Neerincx, and F.M. Jong de, "Arousal and Valence Prediction in Spontaneous Emotional Speech: Felt versus Perceived Emotion," Proc. Ann. Conf. Int'l Speech Comm. Assoc., pp. 2027-2030, 2009.
60. D. Vrakas and I.P. Vlahavas, Artificial Intelligence for Advanced Problem Solving Techniques. IGI Global Snippet, 2008.
61. D. Vukadinovic and M. Pantic, "Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers," Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol. 2, pp. 1692-1698, 2005. (Pubitemid 41667548)
62. J. Wagner, J. Kim, and E. Andre, "From Physiological Signals to Emotions: Implementing and Comparing Selected Methods for Feature Extraction and Classification," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 940-943, 2005. (Pubitemid 44669023)
63. J. Weston, O. Chapelle, A. Elisseeff, B. Schölkopf, and V. Vapnik, "Kernel Dependency Estimation," Technical Report 98, Aug. 2002.
64. M. Wöllmer, B. Schuller, F. Eyben, and G. Rigoll, "Combining Long Short-Term Memory and Dynamic Bayesian Networks for Incremental Emotion-Sensitive Artificial Listening," IEEE J. Selected Topics in Signal Processing, vol. 4, no. 5, pp. 867-881, Oct. 2010.
65. Y.-H. Yang, Y.-C. Lin, Y.-F. Su, and H.H. Chen, "Music Emotion Classification: A Regression Approach," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 208-211, 2007.
66. C. Yu, P.M. Aoki, and A. Woodruff, "Detecting User Engagement in Everyday Conversations," Proc. Eighth Int'l Conf. Spoken Language Processing, 2004.
67. Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.