Emotion representation, analysis and synthesis in continuous space: A survey

2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, FG 2011

Volumn , Issue , 2011, Pages 827-834

  Gunes, Hatice ^a,b   Schuller, Bjöorn ^d   Pantic, Maja ^a,c   Cowie, Roddy ^e  

^a IMPERIAL COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

^c UNIVERSITY OF TWENTE   (Netherlands)

^d TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^e QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

AFFECTIVE COMPUTING; AFFECTIVE STATE; ANALYSIS AND SYNTHESIS; AUTOMATED SYSTEMS; BEST MODEL; CONTINUOUS ANALYSIS; CONTINUOUS SPACES; EMOTION REPRESENTATION; FUZZY BOUNDARY; STATE OF THE ART;

AUTOMATION; BEHAVIORAL RESEARCH; FACE RECOGNITION; MATHEMATICAL MODELS;

GESTURE RECOGNITION;

EID: 79958702587     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/FG.2011.5771357     Document Type: Conference Paper

References (121)

1. D. Grandjean, D. Sander, & K. R. Scherer, "Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization," Consciousness & Cognition, vol. 17, no. 2, pp. 484-495, 2008, Social Cognition, Emotion, & Self-Consciousness.
2. P. Ekman & W.V. Friesen, Unmasking The Face: A Guide To Recognizing Emotions From Facial Clues, Prentice Hall, New Jersey, 1975.
3. J. A. Russell, "A circumplex model of affect," J. of Personality & Social Psychology, vol. 39, pp. 1161-1178, 1980.
4. A. Mehrabian, "Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament," Current Psychology: Developmental, Learning, Personality, Social, vol. 14, pp. 261-292, 1996.
5. K.R. Scherer, A. Schorr, & T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research, Oxford Univ. Press, Oxford/New York, 2001.
6. J. Jia et al., "Emotional audio-visual speech synthesis based on pad," IEEE Trans. on Audio, Speech, & Language Processing, vol. PP, no. 9, pp. 1-1, 2010.
7. H.P. Espinosa, C.A.R. Garcia, & L.V. Pineda, "Features selection for primitives estimation on emotional speech," in Proc. IEEE ICASSP, 2010, pp. 5138-5141.
8. J. R. Fontaine et al., "The world of emotion is not two-dimensional," Psychological Science, vol. 18, pp. 1050-1057, 2007.
9. G. McKeown et al., "The semaine corpus of emotionally coloured character interactions," in Proc. IEEE ICME, 2010, pp. 1079-1084.
10. N. H. Frijda, The emotions, Cambridge Univ. Press, 1986.
11. D. Sander, D. Grandjean, & K. R. Scherer, "A systems approach to appraisal mechanisms in emotion," Neural Netw., vol. 18, no. 4, pp. 317-352, 2005. (Pubitemid 40922644)
12. A. Ortony, G. L. Clore, & A. Collins, The cognitive structure of emotions, Cambridge Univ. Press, Oxford, 1988.
13. C. Bartneck, "Integrating the occ model of emotions in embodied characters," 2002, pp. 39-48.
14. H. Gunes & M. Pantic, "Automatic, dimensional and continuous emotion recognition," Int. J. of Synthetic Emotions, vol. 1, no. 1, pp. 68-99, 2010.
15. R. Dietz & A. Lang, "Affective agents: Effects of agent affect on arousal, attention, liking and learning," in Proc. Cognitive Technology, 1999.
16. G. Chanel, K. Ansari-Asl, & T. Pun, "Valence-arousal evaluation using physiological signals in an emotion recall paradigm," in Proc. IEEE SMC, Oct. 2007, pp. 2662-2667. (Pubitemid 351409969)
17. A. Haag et al., "Emotion recognition using bio-sensors: First steps towards an automatic system," in LNCS 3068, 2004, pp. 36-48. (Pubitemid 38851666)
18. T. Pun et al., "Brain-computer interaction research at the computer vision and multimedia laboratory, univ. of geneva," IEEE Tran. on Neural Systems & Rehabilitation Engineering, vol. 14, pp. 210-213, 2006. (Pubitemid 44150244)
19. R.J. Davidson & N.A. Fox, "Asymmetrical brain activity discriminates between positive and negative affective stimuli in human infants," Science, vol. 218, pp. 1235-1237, 1982. (Pubitemid 13224836)
20. P. Tsiamyrtzis et al., "Imaging facial physiology for the detection of deceit," Int. J. of Computer Vision, 2007.
21. R. Cowie et al., "Feeltrace: An instrument for recording perceived emotion in real time," in Proc. ISCA Workshop on Speech & Emotion, 2000, pp. 19-24.
22. R. Cowie et al., "Emotion recognition in human-computer interaction," IEEE Signal Processing Magazine, vol. 18, pp. 33-80, 2001.
23. R.A. Calvo & S. D'Mello, "Affect detection: An interdisciplinary review of models, methods, and their applications," IEEE Trans. on Affective Computing, vol. 1, no. 1, pp. 18-37, 2010.
24. G. L. Huttar, "Relations between prosodic variables and emotions in normal american english utterances," J. Speech Hearing Res., vol. 11, pp. 481-487, 1968.
25. J. Davitz, The Communication of Emotional Meaning, chapter Auditory correlates of vocal expression of emotional feeling, pp. 101-112, McGraw-Hill, 1964.
26. K. R. Scherer & J. S. Oshinsky, "Cue utilization in emotion attribution from auditory stimuli," Motivation Emotion, vol. 1, pp. 331-346, 1977.
27. J. Trouvain & W. J. Barry, "The prosody of excitement in horse race commentaries," in Proc. ISCA Workshop Speech Emotion, 2000, pp. 86-91.
28. M. Schr̈oder, Speech and Emotion Research: An Overview of Research Frameworks and a Dimensional Approach to Emotional Speech Synthesis, PH.D. dissertation, Univ. of Saarland, Germany, 2003.
29. M. Schr̈oder, D. Heylen, & I. Poggi, "Perception of non-verbal emotional listener feedback," in Speech Prosody, R. Hoffmann & H. Mixdorff, Eds., 2006, pp. 1-4.
30. P. Ekman & W. V. Friesen, "Head and body cues in the judgment of emotion: A reformulation," Perceptual & Motor Skills, vol. 24, pp. 711-724, 1967.
31. M. Chen & J. A. Bargh, "Consequences of automatic evaluation: Immediate behavioral predispositions to approach or avoid the stimulus," Personality & Social Psychology Bulletin, vol. 25, pp. 215-224, 1999.
32. J. Forster & F. Strack, "Influence of overt head movements on memory for valenced words: A case of conceptual-motor compatibility," J. of Personality & Social Psychology, vol. 71, pp. 421-430, 1996. (Pubitemid 126443609)
33. C.S. Carver, "Pleasure as a sign you can attend to something else: Placing positive feelings within a general model of affect," Cognition & Emotion, vol. 17, pp. 241-261, 2003. (Pubitemid 36267018)
34. C.H. Hillman, K.S. Rosengren, & D.P. Smith, "Emotion and motivated behavior: postural adjustments to affective picture viewing," Biological Psychology, vol. 66, pp. 51-62, 2004. (Pubitemid 38281180)
35. R. Cowie et al., "The emotional and communicative significance of head nods and shakes in a naturalistic database," in Proc. LREC Int. Workshop on Emotion, 2010, pp. 42-46.
36. M. Ẅollmer et al., "Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling," in Proc. INTERSPEECH, 2010, pp. 2362-2365.
37. A. Kleinsmith & N. Bianchi-Berthouze, "Recognizing affective dimensions from body posture," in Proc. ACII, 2007, pp. 48-58.
38. A. Kleinsmith, P. R. De Silva, & N. Bianchi-Berthouze, "Recognizing emotion from postures: Cross-cultural differences in user modeling," in Proc. the Conf. on User Modeling, 2005, pp. 50-59.
39. H. K. Meeren, C. C. Van Heijnsbergen, & B. De Gelder, "Rapid perceptual integration of facial expression and emotional body language," Proc. the National Academy of Sciences, vol. 102, pp. 16518-16523, 2005. (Pubitemid 41688946)
40. S. Buisine et al., "Perception of blended emotions: From video corpus to expressive agent," in Proc. IVA, 2006, pp. 93-106. (Pubitemid 44532606)
41. Z. Khalili & M.H. Moradi, "Emotion recognition system using brain and peripheral signals: Using correlation dimension to improve the results of eeg," in Proc. Int. Joint Conf. on Neural Networks, 2009, pp. 1571-1575.
42. G. Chanel et al., "Short-term emotion assessment in a recall paradigm," Int. J. of Human-Computer Studies, vol. 67, no. 8, pp. 607-627, 2009.
43. D. Kulic & E. A. Croft, "Affective state estimation for human-robot interaction.," IEEE Trans. on Robotics, vol. 23, no. 5, pp. 991-1000, 2007. (Pubitemid 47577975)
44. P.J. Lang, The cognitive psychophysiology of Emotion: Anxiety and the anxiety disorders, Lawrence Erlbaum, NJ, 1985.
45. G. Laurans, P. Desmet, & P. Hekkert, "The emotion slider: A selfreport device for the continuous measurement of emotion," in Proc. ACII Workshops, 2009, pp. 1-6.
46. J.M. Kessens et al., "Perception of synthetic emotion expressions in speech: Categorical and dimensional annotations," in Proc. ACII Workshops, 2009, pp. 1-5.
47. E. Douglas-Cowie et al., "The humaine database: addressing the needs of the affective computing community," in Proc. ACII, 2007, pp. 488-500.
48. M. Grimm, K. Kroschel, & S. Narayanan, "The vera am mittag german audio-visual emotional speech database.," in Proc. IEEE ICME, 2008, pp. 865-868.
49. B. Schuller et al., "Special issue on naturalistic affect resources for system building and evaluation," IEEE Tran. on Affective Computing, 2011.
50. H. Gunes, M. Piccardi, & M. Pantic, Affective Computing: Focus on Emotion Expression, Synthesis, and Recognition, Ch. From the Lab to the Real World: Affect Recognition using Multiple Cues and Modalities, pp. 185-218, I-Tech Educ. & Publishing, 2008.
51. Z. Zeng et al., "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Tran. on Pattern Analysis & Machine Intelligence, vol. 31, pp. 39-58, 2009.
52. Y. Gu et al., "Emotion-aware technologies for consumer electronics," in Proc. IEEE Int. Symp. on Consumer Electronics, 2008, pp. 1-4.
53. R. Khosrowabadi et al., "Eeg-based emotion recognition using selforganizing map for boundary detection," in Proc. ICPR, 2010, pp. 4242-4245.
54. C.A. Frantzidis et al., "On the classification of emotional biosignals evoked while viewing affective pictures: An integrated data-miningbased approach for healthcare applications," IEEE Trans. on Information Techn. in Biomedicine, vol. 14, no. 2, pp. 309-318, 2010.
55. J. Kim & E. Andre, "Emotion recognition based on physiological changes in music listening," IEEE Trans. on Pattern Analysis & Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, 2008.
56. B. Schuller et al., "Acoustic emotion recognition: A benchmark comparison of performances," in Proc. IEEE ASRU, 2009.
57. M. Ẅollmer et al., "Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening," IEEE J. of Selected Topics in Signal Processing, vol. 4, no. 5, pp. 867-881, Oct. 2010.
58. M.Ẅollmer et al., "Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies," in Proc. Interspeech, 2008, pp. 597-600.
59. M. Grimm & K. Kroschel, "Emotion estimation in speech using a 3d emotion space concept," in Proc. IEEE Automatic Speech Recog. & Understanding Workshop, 2005, pp. 381-385.
60. B. Schuller et al., "The interspeech 2010 paralinguistic challenge," in Proc. INTERSPEECH, 2010, pp. 2794-2797.
61. M.A. Nicolaou, H. Gunes, & M. Pantic, "Audio-visual classification and fusion of spontaneous affective data in likelihood space," in Proc. Int. Conf. on Pattern Recognition, 2010, pp. 3695-3699.
62. D. McDuff et al., "Affect valence inference from facial action unit spectrograms," in Proc. IEEE CVPR Workshops, 2010, pp. 17-24.
63. D. Glowinski et al., "Technique for automatic emotion recognition by body gesture analysis," in Proc. IEEE CVPR Workshops, 2008, pp. 1-6.
64. S. Ioannou et al., "Emotion recognition through facial expression analysis based on a neurofuzzy method," J. of Neural Networks, vol. 18, pp. 423-435, 2005. (Pubitemid 40922649)
65. H. Gunes & M. Pantic, "Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners," in Proc. of IVA, 2010, pp. 371-377.
66. M. Kipp & J.-C. Martin, "Gesture and emotion: Can basic gestural form features discriminate emotions?," in Proc. ACII Workshops, 2009, pp. 1-8.
67. M.A. Nicolaou, H. Gunes, & M. Pantic, "Output-associative rvm regression for dimensional and continuous emotion prediction," in Proc. of IEEE FG, 2011.
68. M.A. Nicolaou, H. Gunes, & M. Pantic, "Automatic segmentation of spontaneous data using dimensional labels from multiple coders," in Proc. LREC Int. Workshop on Multimodal Corpora, 2010, pp. 43-48.
69. S. Arifin & P. Cheung, "Affective level video segmentation by utilizing the pleasure-arousal-dominance information," IEEE Trans. on Multimedia, vol. 10, no. 7, pp. 1325-1341, 2008.
70. B.R. Nhan & T. Chau, "Classifying affective states using thermal infrared imaging of the human face," IEEE Tran. on Biomedical Engineering, vol. 57, no. 4, pp. 979-987, 2010.
71. A. Merla & G.L. Romani, "Thermal signatures of emotional arousal: a functional infrared imaging study," in Proc. Int. Conf. of the IEEE Engineering in Medicine & Biology Society, 2007, pp. 247-249.
72. G. Caridakis et al., "Modeling naturalistic affective states via facial and vocal expressions recognition," in Proc. ACM ICMI, 2006, pp. 146-154. (Pubitemid 47113445)
73. I. Kanluan, M. Grimm, & K. Kroschel, "Audio-visual emotion recognition using an emotion recognition space concept," Proc. European Signal Processing Conf., 2008.
74. F. Eyben et al, "String-based Audiovisual Fusion of Behavioural Events for the Assessment of Dimensional Affect," in Proc. IEEE FG, 2011.
75. S.W. Gilroy et al., "Pad-based multimodal affective fusion," in Proc. ACII Workshops, 2009, pp. 1-8.
76. G.G. Berntson et al., "Heart rate variability:origins, methods, and interpretive caveats," Psychophysiology, vol. 34, no. 6, pp. 623.
77. L. Salahuddin et al., "Ultra short term analysis of heart rate variability for monitoring mental stress in mobile settings," in Proc. IEEE Int. Conf. of EMBS, 2007, pp. 39-48.
78. A. Nakasone, H. Prendinger, & M. Ishizuka, "Emotion recognition from electromyography and skin conductance," in Proc. Int. Workshop on Biosignal Interpretation, 2005, pp. 219-222.
79. S. Petridis et al., "Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities," in Proc. ACM ICMI, 2009, pp. 23-30.
80. S. Pammi, M. Charfuelan, & M. Schr̈oder, "Multilingual voice creation toolkit for the mary tts platform," in Proc. LREC, 2010.
81. E. Eide et al., "A corpus-based approach to ¡ahem/¿ expressive speech synthesis," in Proc. IEEE Speech Synthesis Workshop, 2002, pp. 79-84.
82. S. Mori, T. Moriyama, & S. Ozawa, "Emotional speech synthesis using subspace constraints in prosody," in Proc. IEEE ICME, 2006, pp. 1093-1096. (Pubitemid 46679911)
83. M. Bulut, Sungbok Lee, & S. Narayanan, "A statistical approach for modeling prosody features using pos tags for emotional speech synthesis," in Proc. IEEE ICASSP, 2007, vol. 4, pp. 1237-1240.
84. Y. Qin, X. Zhang, & H. Ying, "A hmm-based fuzzy affective model for emotional speech synthesis," in Proc. Int. Conf. on Signal Processing Systems, 2010, vol. 3, pp. 525-528.
85. M. Bulut, Sungbok Lee, & S. Narayanan, "Recognition for synthesis: Automatic parameter selection for resynthesis of emotional speech from neutral speech," in Proc. IEEE ICASSP, 2008, pp. 4629-4632.
86. J. Tao, Y. Kang, & A. Li, "Prosody conversion from neutral speech to emotional speech," IEEE Trans. on Audio, Speech, & Language Processing, vol. 14, no. 4, pp. 1145-1154, 2006.
87. S. Lv, S. Wang, & X. Wang, "Emotional speech synthesis by xml file using interactive genetic algorithms," in Proc. ACM/SIGEVO Summit on Genetic & Evolutionary Computation, 2009.
88. F. Burkhardt & J. Stegmann, "Emotional speech synthesis: Applications, history and possible future," in Proc. ESSV, 2009.
89. E. Bevacqua & C. Pelachaud, "Expressive audio-visual speech," Computer Animation & Virtual Worlds, vol. 15, no. 3-4, pp. 297-304, 2004. (Pubitemid 41108008)
90. M. Schr̈oder, "Emotional speech synthesis:a review," in Proc. Eurospeech, 2001, vol. 1, pp. 561-564.
91. H. Tang et al., "Humanoid audiovisual avatar with emotive text-tospeech synthesis," IEEE Trans. on Multimedia, vol. 10, no. 6, pp. 969-981, 2008.
92. Z. Ruttkay, H. Noot, & P. Hagen, "Emotion disc and emotion squares: Tools to explore the facial expression space," Computer Graphics Forum, vol. 22, pp. 49-53, 2003.
93. I. Albrecht et al., "Mixed feelings: expression of non-basic emotions in a musclebased talking head," Virtual Reality, vol. 8, pp. 201-212, 2005. (Pubitemid 41386797)
94. C. Pelachaud & M. Bilvi, "Computational model of believable conversational agents," in LNCS 2650, 2003, pp. 300-317.
95. K. C.Wassermann, K. Eng, & P. F. M. J. Verschure, "Live soundscape composition based on synthetic emotions," IEEE Multimedia, vol. 10, pp. 82-90, 2003.
96. C. Busso et al., "Rigid head motion in expressive speech animation: Analysis and synthesis," IEEE Trans. on Audio, Speech, & Language Processing, vol. 15, no. 3, pp. 1075-1086, 2009.
97. H. Boukricha et al., "Pleasure-arousal-dominance driven facial expression simulation," in Proc. ACII Workshops, 2009, pp. 1-7.
98. X. Shen, X. Fu, & Y. Xuan, "Do different emotional valences have same effects on spatial attention?," in Proc. of Int. Conf. on Natural Computation, 2010, vol. 4, pp. 1989-1993.
99. D. Heylen et al., "Social agents: The first generations," in Proc. ACII Workshops, 2009, pp. 1-7.
100. M. Schr̈oder et al., "A demonstration of audiovisual sensitive artificial listeners," in Proc. ACII, 2009, vol. 1, pp. 263-264.
101. H. Buschmeier, K. Bergmann, & S. Kopp, "Adaptive expressiveness virtual conversational agents that can align to their interaction partner," in Proc. Int. Conf. on Autonomous Agents & Multiagent Systems, 2010, pp. 91-98.
102. C. Pelachaud, "Modelling multimodal expression of emotion in a virtual agent," Philosophical Trans. of Royal Society B Biological Science, vol. 364, pp. 3539-3548, 2009.
103. E. Bevacqua et al., "Facial feedback signals for ecas," in Proc. Artificial & Ambient Intelligence, 2007, pp. 1-7.
104. M. Schr̈oder, "The semaine api: Towards a standards-based framework for building emotion-oriented systems," Advances in Human-Machine Interaction, vol. 2010, pp. 1-21, 2010.
105. F. Eyben, M. Ẅollmer, & B. Schuller, "opensmile-the munich versatile and fast open-source audio feature extractor," in Proc. ACM Multimedia, 2010, pp. 1459-1462.
106. T. Taleb, D. Bottazzi, & N. Nasser, "A novel middleware solution to improve ubiquitous healthcare systems aided by affective information," IEEE Trans. on Information Technology in Biomedicine, vol. 14, no. 2, pp. 335-349, 2010.
107. T.S. Huang et al., "Sensitive talking heads," IEEE Signal Processing Magazine, vol. 26, pp. 67-72, 2009.
108. A. Beck, L. Canamero, & K.A. Bard, "Towards an affect space for robots to display emotional body language," in Proc. IEEE RO-MAN, 2010, pp. 464-469.
109. M. Karg et al., "Towards mapping emotive gait patterns from human to robot," in Proc. IEEE RO-MAN, 2010, pp. 258-263.
110. M. Mihelj, D. Novak, & M. Munih, "Emotion-aware system for upper extremity rehabilitation," in Proc. Int. Conf. on Virtual Rehabilitation, 2009, pp. 160-165.
111. T.-C. Tsai, J.-J. Chen, & W.-C. Lo, "Design and implementation of mobile personal emotion monitoring system," in Proc. Int. Conf. on Mobile Data Management, 2009, pp. 430-435.
112. B. Grundlehner et al., "The design and analysis of a real-time, continuous arousal monitor," in Proc. Int. Workshop on Wearable & Implantable Body Sensor Networks, 2009, pp. 156-161.
113. U. Faghihi et al., "How emotional mechanism helps episodic learning in a cognitive agent," in Proc. IEEE Symp. on Intelligent Agents, 2009, pp. 23-30.
114. F. Eyben, M. Ẅollmer, T. Poitschke, B. Schuller, C. Blaschke, B. F̈arber, & N. Nguyen-Thien, "Emotion on the road-necessity, acceptance, and feasibility of affective computing in the car," Advances in Human-Machine Interaction, vol. 2010, pp. 1-17, 2010.
115. K. Sun et al., "An improved valence-arousal emotion space for video affective content representation and recognition," in Proc. IEEE ICME, 2009, pp. 566-569.
116. J.J.M. Kierkels, M. Soleymani, & T. Pun, "Queries and tags in affectbased multimedia retrieval," in Proc. IEEE ICME, 2009, pp. 1436-1439.
117. M. Soleymani, J. Davis, & T. Pun, "A collaborative personalized affective video retrieval system," in Proc. ACII Workshops, 2009, pp. 1-2.
118. M. Rehm & M. Wissner, "Gamble a multiuser game with an embodied conversational agent," in LNCS 3711, 2005, pp. 180-191.
119. "Affectiva's homepage: http://www.affectiva.com/," 2010.
120. C. Allen, W. Wallach, & I. Smit, "Why machine ethics?," IEEE Intelligent Systems, vol. 21, pp. 12-17, 2006.
121. A. Beavers, "Special issue on ethics & affective computing," IEEE Trans. on Affective Computing, 2011.