Speech-based recognition of self-reported and observed emotion in a dimensional space

Speech Communication

Volumn 54, Issue 9, 2012, Pages 1049-1063

  Truong, Khiet P ^a   Van Leeuwen, David A ^b   De Jong, Franciska M G ^a  

^a UNIVERSITY OF TWENTE   (Netherlands)

^b RADBOUD UNIVERSITY NIJMEGEN   (Netherlands)

Author keywords

Affective computing; Audiovisual database; Automatic emotion recognition; Emotion annotation; Emotion database; Emotion elicitation; Emotion perception; Emotional speech; Support Vector Regression; Videogames

Indexed keywords

AFFECTIVE COMPUTING; AUDIO-VISUAL DATABASE; EMOTION ANNOTATION; EMOTION ELICITATION; EMOTION RECOGNITION; EMOTIONAL SPEECH; SUPPORT VECTOR REGRESSION (SVR); VIDEO GAME;

SPEECH RECOGNITION; VECTOR SPACES;

BEHAVIORAL RESEARCH;

EID: 84862851965     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2012.04.006     Document Type: Article

References (61)

1. Ang, J.; Dhillon, R.; Krupski, A.; Shriberg, E.; Stolcke, A.; 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In: Proceedings of the International Conference on Spoken Language Processing (ICSLP 2002), pp. 2037-2040.
2. Auberge, V.; Audibert, N.; Rilliard, A.; 2006. Auto-annotation: an alternative method to label expressive corpora. In: Proc. Fifth Internat. Conf. on Language Resources and Evaluation (LREC 2006).
3. R. Banse, and K.R. Scherer Acoustic profiles in vocal emotion expression J. Pers. Soc. Psychol. 70 1996 614 636
4. Batliner, A.; Fischer, K.; Huber, R.; Spilker, J.; Nöth, E.; 2000. Desperately seeking emotions: actors, wizards, and human beings. In: Cowie, R.; Douglas-Cowie, E.; Schröder, M. (Eds.), In: Proc. ISCA Workshop on Speech and Emotion: A Conceptual Framework for Research, pp. 195-200.
5. Batliner, A.; Steidl, S.; Schuller, B.; Seppi, D.; Laskowski, K.; Vogt, T.; Devillers, L.; Vidrascu, L.; Amir, N.; Kessous, L.; Aharonson, V.; 2006. Combining efforts for improving automatic classification of emotional user states. In: Language Technologies (IS-LTC), pp. 240-245.
6. Biersack, S.; Kempe, V.; 2005. Tracing vocal expression of emotion along the speech chain: do listeners perceive what speakers feel? In: Proc. ISCA Workshop on Plasticity in Speech Perception (PSP2005), pp. 211-214.
7. Boersma, P.; Weenink, D.; 2009. Praat: doing phonetics by computer (Version 5.1.07). [Computer Program]. Retrieved 16.06.09.
8. Busso, C.; Narayanan, S.S.; 2008. The expression and perception of emotions: comparing assessments of self versus others. In: Proc. Interspeech 2008, pp. 257-260.
9. Chang, C.-C.; Lin, C.-J.; 2001. LIBSVM: a library for support vector machines. .
10. R. Cowie, and R.R. Cornelius Describing the emotional states that are expressed in speech Speech Commun. 40 2003 5 32
11. Cowie, R.; Douglas-Cowie, E.; Savvidou, S.; McMahon, E.; Sawey, M.; Schroeder, M.; 2000. FEELTRACE: an instrument for recording perceived emotion in real time. In: Proc. ISCA ITRW on Speech and Emotion, pp. 19-24.
12. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor Emotion recognition in human-computer interaction IEEE Signal Process. Mag. 18 1 2001 32 80
13. Dellaert, F.; Polzin, T.; Waibel, A.; 1996. Recognizing emotion in speech. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP 1996), pp. 1970-1973.
14. Den Uyl, M.J.; Van Kuilenburg, H.; 2005. The facereader: online facial expression recognition. In: Proc. Measuring Behavior, pp. 589-590.
15. Devillers, L.; Lamel, L.; Vasilescu, I.; 2003. Emotion detection in task-oriented spoken dialogues. In: Proc. IEEE Internat. Conf. on Multimedia and Expo (ICME'03), pp. 549-552.
16. L. Devillers, L. Vidrascu, and L. Lamel Challenges in real-life emotion annotation and machine learning based detection Neural Networks 18 2005 407 422
17. Douglas-Cowie, E.; Devillers, L.; Martin, J.-C.; Cowie, R.; Davvidou, S.; Abrillian, S.; Cox, C.; 2005. Multimodal databases of everyday emotion: facing up to complexity. In: Proc. Interspeech 2005, pp. 813-816.
18. Ekman, P.; 1972. Universals and cultural differences in facial expressions of emotion. In: Cole, J. (Ed.), Nebraska Symposium on Motivation, pp. 207-283.
19. P. Ekman, and W. Friesen Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions 1975 Prentice Hall, Inc. Englewood Cliffs, New Jersey
20. F. Eyben, M. Wöllmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie On-line emotion recognition in a 3-d activation-valence-time continuum using acoustic and linguistic cues J. Multimodal User Interf. 3 2010 7 19
21. Giannakopoulos, T.; Pikrakis, A.; Theodoridis, S.; 2009. A dimensional approach to emotion recognition of speech from movies. In: Proc. IEEE Internat. Conf. on Acoustics, Speech and Signal Processing (ICASSP'09), pp. 65-68.
22. M. Grimm, K. Kroschel, E. Mower, and S. Narayanan Primitives-based evaluation and estimation of emotions in speech Speech Commun. 49 2007 787 800
23. Grimm, M.; Kroschel, K.; Narayanan, S.; 2007b. Support Vector Regression for automatic recognition of spontaneous emotions in speech. In: Proc. IEEE Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP 2007), pp. 1085-1088.
24. Gunes, H.; Schuller, B.; Pantic, M.; Cowie, R.; 2011. Emotion representation, analysis and synthesis in continuous space: a survey. In: Proc. IEEE Internat. Conf. on Automatic Face & Gesture Recognition and Workshops (FG2011), pp. 827-834.
25. A. Hanjalic, and L.-Q. Xu Affective video content representation and modeling IEEE Trans. Multimedia 7 2005 143 154
26. Joachims, T.; 1998. Text categorization with support vector machines: learning with many relevant features. In: Proc. 10th European Conf. on Machine Learning (ECML-98), pp. 137-142.
27. T. Johnstone, C.M. van Reekum, K. Hird, K. Kirsner, and K.R. Scherer Affective speech elicited with a computer game Emotion 5 4 2005 513 518
28. Kim, J.; André, E.; Rehm, M.; Vogt, T.; Wagner, J.; 2005. Integrating information from speech and physiological signals to achieve emotional sensitivity. In: Proc. Interspeech, pp. 809-812.
29. Kwon, O.-W.; Chan, K.; Hao, J.; Lee, T.-W.; 2003. Emotion recognition by speech signals. In: Proc. Eurospeech, pp. 125-128.
30. P. Lang The emotion probe Amer. Psychol. 50 1995 371 385
31. Lazarro, N.; 2004. Why We Play Games: Four Keys to More Emotion Without Story.
32. Lee, C.M.; Narayanan, S.; Pieraccini, R.; 2002. Classifying emotions in human-machine spoken dialogs. In: Proc. IEEE Internat. Conf. on Multimedia and Expo (ICME '02), pp. 737-740.
33. Liscombe, J.; Venditti, J.; Hirschberg, J.; 2003. Classifying subject ratings of emotional speech using acoustic features. In: Proc. Eurospeech, pp. 725-728.
34. Mower, E.; Mataric, M.J.; Narayanan, S.S.; 2009. Evaluating evaluators: a case study in understanding the benefits and pitfalls of multi-evaluator modeling. In: Proc. Interspeech 2009, pp. 1583-1586.
35. Nicolaou, M.A.; Gunes, H.; Pantic, M.; 2010. Automatic segmentation of spontaneous data using dimensional labels from multiple coders. In: Proc. Internat. Workship on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, pp. 43-48.
36. M.A. Nicolaou, H. Gunes, and M. Pantic Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space IEEE Trans. Affect. Comput. 2 2011 92 105
37. T.L. Nwe, S.W. Foo, and L.C. De Silva Speech Commun. 41 2003 603 623
38. Petrushin, V.A.; 1999. Emotion in speech: recognition and application to call centers. In: Proc. 1999 Conf. on Artificial Neural Networks in Engineering (ANNIE'99).
39. Polzin, T.; Waibel, A.; 1998. Detecting emotions in speech. In: Proc. Cooperative Multimodal Communication (CMC'98).
40. N. Ravaja, T. Saari, M. Turpeinen, J. Laarni, M. Salminen, and M. Kivikangas Spatial presence and emotions during video game playing: does it matter with whom you play? Presence: Teleoper. Virtual Environ. 15 2006 381 392
41. J.A. Russell A circumplex model of affect J. Pers. Soc. Psychol. 39 1980 1161 1178
42. G. Salton, and C. Buckley Term-weighting approaches in automatic text retrieval Inform. Process. Manage. 24 5 1988 513 523
43. Scherer, K.R.; 2010. The component process model: architecture for a comprehensive computational model of emergent emotion. In: Scherer, K.R.; Bänziger, T.; Roesch, E. (Eds.), Blueprint for Affective Computing: A Sourcebook, pp. 47-70.
44. H. Schlosberg Psychol. Rev. 61 1954 81 88
45. Schuller, B.; Rigoll, G.; Lang, M.; 2003. Hidden Markov model-based speech emotion recognition. In: Proc. IEEE Internat. Conf. on Acoustics, Speech, and Signal Processing (ICASSP 2003), pp. 1-4.
46. A.J. Smola, and B. Scholkopf A tutorial on support vector regression Stat. Comput. 14 3 2004 199 222
47. Tato, R.; Santos, R.; Kompe, R.; Pardo, J.M.; 2002. Emotional space improves emotion recognition. In: Proc. Internat. Conf. on Spoken Language Processing (ICSLP 2002), pp. 2029-2032.
48. Truong, K.P.; Neerincx, M.A.; Van Leeuwen, D.A.; 2008. Assessing agreement of observer- and self-annotations in spontaneous multimodal emotion data. In: Proc. Interspeech 2008, pp. 381-321.
49. Truong, K.P.; Raaijmakers, S.; 2008. Automatic recognition of spontaneous emotions in speech using acoustic and lexical features. In: Proc. Fifth Joint Workshop on Machine Learning and Multimodal Interaction (MLMI 2008), pp. 161-172.
50. Truong, K.P.; Van Leeuwen, D.A.; Neerincx, M.A.; De Jong, F.M.G.; 2009. Arousal and valence prediction in spontaneous emotional speech: felt versus perceived emotion. In: Proc. Interspeech, pp. 2027-2030.
51. V.N. Vapnik The Nature of Statistical Learning Theory 2002 Springer-Verlag New York
52. Ververidis, D.; Kotropoulos, C.; 2005. Emotional speech classification using gaussian mixture models and the sequential floating forward selection algorithm. In: Proc. IEEE Internat. Conf. on Multimedia and Expo (ICME 2005), pp. 1500 -1503.
53. D. Ververidis, and C. Kotropoulos Emotional speech recognition: resources, features, and methods Speech Commun. 48 9 2006 1162 1181
54. Wang, N.; Marsella, S.; 2006. Introducing EVG: an emotion evoking game. In: Proc. Internat. Conf. on Interactive Virtual Agents (IVA 2006), pp. 282 - 291.
55. C.E. Williams, and K.N. Stevens Emotions and speech: some acoustical correlates J. Acoust. Soc. Amer. 52 4B 1972 1238 1250
56. Wöllmer, M.; Eyben, F.; Reiter, S.; Schuller, B.; Cox, C.; Douglas-Cowie, E.; Cowie, R.; 2008. Abandoning emotion classes - towards continuous emotion recognition with modeling of long-range dependencies. In: Proc. Interspeech, pp. 597-600.
57. Wöllmer, M.; Eyben, F.; Schuller, B.; Douglas-Cowie, E.; Cowie, R.; 2009. Data-driven clustering in emotional space for affect recognition using discriminatively trained lstm networks. In: Proc. Interspeech, pp. 1595-1598.
58. W. Wundt Grundzüge der physiologischen Psychologie, [Fundamentals of pshysiological psychology] 1874/1905 Engelmann Leipzig
59. Yildirim, S.; Lee, C.M.; Lee, S.; Potamianos, A.; Narayanan, S.S.; 2005. Detecting politeness and frustration state of a child in a conversational computer game. In: Proc. Interspeech 2005, pp. 2209-2212.
60. Yu, C.; Aoki, P.; Woodruff, A.; 2004. Detecting user engagement in everyday conversations. In: Proc. Interspeech, pp. 1329-1332.
61. Zeng, Z.; Zhang, Z.; Pianfetti, B.; Tu, J.; Huang, T.S.; 2005. Audio-visual affect recognition in activation-evaluation space. In: Proc. IEEE Internat. Conf. on Multimedia and Expo (ICME'05), pp. 828-831.