Whodunnit - Searching for the most important feature types signalling emotion-related user states in speech

Computer Speech and Language

Volumn 25, Issue 1, 2011, Pages 4-28

  Batliner, Anton ^a   Steidl, Stefan ^a   Schuller, Björn ^b   Seppi, Dino ^c   Vogt, Thurid ^d   Wagner, Johannes ^d   Devillers, Laurence ^e   Vidrascu, Laurence ^e   Aharonson, Vered ^f   Kessous, Loic ^g   Amir, Noam ^g  

^a UNIVERSITY OF ERLANGEN NUREMBERG   (Germany)

^b TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^c ITC IRST   (Italy)

^d UNIVERSITY OF AUGSBURG   (Germany)

^e CNRS   (France)

^f AFEKA TEL AVIV ACADEMIC COLLEGE OF ENGINEERING   (Israel)

^g TEL AVIV UNIVERSITY   (Israel)

Author keywords

Automatic classification; Emotion; Feature selection; Feature types

Indexed keywords

AUTOMATIC CLASSIFICATION; EMOTION FEATURE; FEATURE SELECTION; FEATURE TYPES; FEATURE VECTORS; LINGUISTIC FEATURES; PET ROBOTS;

AUTOMATIC INDEXING; LINGUISTICS;

FEATURE EXTRACTION;

EID: 77955412961     PISSN: 08852308     EISSN: 10958363     Source Type: Journal    
DOI: 10.1016/j.csl.2009.12.003     Document Type: Article

References (87)

1. R. Banse, and K.R. Scherer Acoustic profiles in vocal emotion expression Journal of Personality and Social Psychology 70 3 1996 614 636
2. A. Batliner, R. Kompe, A. Kießling, M. Mast, H. Niemann, and E. Nöth M = Syntax + Prosody: a syntactic-prosodic labelling scheme for large spontaneous speech databases Speech Communication 25 4 1998 193 222
3. Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E., 2000a. Desperately seeking emotions: actors, wizards, and human beings. In: Proc. ISCA Workshop on Speech and Emotion, Newcastle, Northern Ireland, pp. 195-200.
4. A. Batliner, R. Huber, H. Niemann, E. Nöth, J. Spilker, and K. Fischer The recognition of emotion W. Wahlster, Verbmobil: Foundations of Speech-to-Speech Translations 2000 Springer Berlin 122 130
5. A. Batliner, K. Fischer, R. Huber, J. Spilker, and E. Nöth How to find trouble in communication Speech Communication 40 2003 117 143
6. Batliner, A., Zeissler, V., Frank, C., Adelhardt, J., Shi, R.P., Nöth, E., 2003b. We are not amused - but how do you know? User states in a multi-modal dialogue system. In: Proc. Interspeech, Geneva, pp. 733-736.
7. Batliner, A., Hacker, C., Steidl, S., Nöth, E., Haas, J., 2004. From emotion to interaction: lessons from real human-machine-dialogues. In: Affective Dialogue Systems, Proc. a Tutorial and Research Workshop, Kloster Irsee, pp. 1-12.
8. Batliner, A., Steidl, S., Hacker, C., Nöth, E., Niemann, H., 2005. Tales of tuning - prototyping for automatic classification of emotional user states. In: Proc. Interspeech, Lisbon, pp. 489-492.
9. Batliner, A., Steidl, S., Schuller, B., Seppi, D., Laskowski, K., Vogt, T., Devillers, L., Vidrascu, L., Amir, N., Kessous, L., Aharonson, V., 2006. Combining efforts for improving automatic classification of emotional user states. In: Proc. IS-LTC 2006, Ljubliana, pp. 240-245.
10. Batliner, A., Steidl, S., Nöth, E., 2007a. Laryngealizations and emotions: how many babushkas? In: Proc. International Workshop on Paralinguistic Speech - Between Models and Data (ParaLing'07), Saarbrücken, pp. 17-22.
11. Batliner, A., Steidl, S., Schuller, B., Seppi, D., Vogt, T., Devillers, L., Vidrascu, L., Amir, N., Kessous, L., Aharonson, V., 2007b. The Impact of F0 extraction errors on the classification of prominence and emotion. In: Proc. ICPhS, Saarbrücken, pp. 2201-2204.
12. Batliner, A., Schuller, B., Schaeffler, S., Steidl, S., 2008a. Mothers, adults, children, pets - towards the acoustics of intimacy. In: Proc. ICASSP 2008, Las Vegas, pp. 4497-4500.
13. A. Batliner, S. Steidl, C. Hacker, and E. Nöth Private emotions vs. social interaction - a data-driven approach towards analysing emotions in speech User Modeling and User-Adapted Interaction 18 2008 175 206
14. Batliner, A., Seppi, D., Steidl, S., Schuller, B., in press. Segmenting into adequate units for automatic recognition of emotion-related episodes: a speech-based approach. Advances in Human-Computer Interaction.
15. Blomberg, M., Elenius, D., 2003. Collection and recognition of children's speech in the PF-Star project. In: Proc. Fonetik 2003, Umeå, Sweden, pp. 81-84.
16. Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W., Weiss, B., 2005. A database of German emotional speech. In: Proc. Interspeech, Lisbon, pp. 1517-1520.
17. Chuang, Z.-J., Wu, C.-H., 2004. Emotion recognition using acoustic features and textual content. In: Proc. ICME, Taipei, pp. 53-56.
18. R. Cowie, and R.R. Cornelius Describing the emotional states that are expressed in speech Speech Communication 40 1-2 2003 5 32
19. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor Emotion recognition in human-computer interaction IEEE Signal Processing Magazine 18 1 2001 32 80
20. Cowie, R., Sussman, N., Ben-Ze'ev, A., in press. Emotions: concepts and definitions. In: Petta, P. (Ed.), HUMAINE handbook on emotion. Springer.
21. M. Davies, and J.L. Fleiss Measuring agreement for multinomial data Biometrics 38 1982 1047 1051
22. Dellaert, F., Polzin, T., Waibel, A., 1996. Recognizing emotion in speech. In: Proc. ICSLP, Philadelphia, pp. 1970-1973.
23. F. de Rosis, A. Batliner, N. Novielli, and S. Steidl 'You are Sooo Cool, Valentina!' recognizing social attitude in speech-based dialogues with an ECA A. Paiva, R. Prada, R.W. Picard, Affective Computing and Intelligent Interaction 2007 Springer Berlin-Heidelberg 179 190
24. Devillers, L., Vidrascu, L., 2004. Reliability of lexical and prosodic cues in two real-life spoken dialog corpora. In: LREC, Lisbon.
25. L. Devillers, L. Vidrascu, and L. Lamel Challenges in real-life emotion annotation and machine learning based detection Neural Networks 18 2005 407 422
26. P. Ekman Basic emotions T. Dalgleish, M. Power, Handbook of Cognition and Emotion 1999 John Wiley New York 301 320
27. Engberg, I.S., Hansen, A.V., Andersen, O., Dalsgaard, P., 1997. Design, recording and verification of a Danish emotional speech database. In: Proc. Eurospeech, Rhodes, pp. 1695-1698.
28. D. Erickson, K. Yoshida, C. Menezes, A. Fujino, T. Mochida, and Y. Shibuya Exploratory study of some acoustic and articulatory characteristics of sad speech Phonetica 63 2004 1 25
29. H. Eysenck The concept of statistcal significance and the controversy about one-tailed tests Psychological Review 67 1960 269 271
30. Fiscus, J., 1997. A post-processing system to yield reduced word error rates: recognizer output voting error reduction (ROVER). In: Proc. ASRU, Santa Barbara, pp. 347-352.
31. J. Fleiss, J. Cohen, and B. Everitt Large sample standard errors of kappa and weighted kappa Psychological Bulletin 72 5 1969 323 327
32. R. Frick Communicating emotion: the role of prosodic features Psychological Bulletin 97 1985 412 429
33. Giuliani, D., Gerosa, M., 2003. Investigating recognition of children's speech. In: Proc. ICASSP 2003, Hong Kong, vol. 2, pp. 137-140.
34. C. Gobl, and A. NíChasaide The role of voice quality in communicating emotion, mood and attitude Speech Communication 40 2003 189 212
35. M. Grimm, K. Kroschel, H. Harris, C. Nass, B. Schuller, G. Rigoll, and T. Moosmayr On the necessity and feasibility of detecting a driver's emotional state while driving A. Paiva, R. Prada, R.W. Picard, Affective Computing and Intelligent Interaction 2007 Springer Berlin-Heidelberg 126 138
36. Hall, M.A., 1998. Correlation-Based Feature Selection for Machine Learning. Ph.D. Thesis, Waikato University, Department of Computer Science, Hamilton, NZ.
37. Hermansky, H., Sharma, S., 1998. Traps - classifiers of temporal patterns. In: Proc. ICSLP, Sydney, pp. 1003-1006.
38. M. Holodynski, and W. Friedlmeier Development of Emotions and Emotion Regulation 2006 Springer New York
39. Hu, H., Xu, M.-X., Wu, W., 2007. GMM supervector based SVM with spectral features for speech emotion recognition. In: Proc. Interspeech, Antwerp, pp. 413-416.
40. Inanoglu, Z., Caneel, R., 2005. Emotive alert: HMM-based emotion detection in voicemail messages. In: Proc. 10th International Conference on Intelligent User Interfaces, San Diego, California, pp. 251-253.
41. A. Jain, and D. Zongker Feature selection: evaluation, application and small sample performance PAMI 19 2 1997 153 158
42. T. Joachims Text categorization with support vector machines: learning with many relevant features C. Nédellec, C. Rouveirol, Proc. ECML-98, 10th European Conference on Machine Learning 1998 Springer Heidelberg, Chemnitz 137 142
43. A. Kießling Extraktion und Klassifikation prosodischer Merkmale in der automatischen Sprachverarbeitung. Berichte aus der Informatik 1997 Shaker Aachen
44. D. Küstner, R. Tato, T. Kemp, and B. Meffert Towards real life applications in emotion recognition E. André, L. Dybkiaer, W. Minker, P. Heisterkamp, Affective Dialogue Systems, Proceedings of a Tutorial and Research Workshop Lecture Notes in Artificial Intelligence vol. 3068 2004 Springer Berlin 25 35
45. Kwon, O.-W., Chan, K., Hao, J., Lee, T.-W., 2003. Emotion recognition by speech signals. In: Proc. Interspeech, pp. 125-128.
46. C.M. Lee, and S.S. Narayanan Toward detecting emotions in spoken dialogs IEEE Transactions on Speech and Audio Processing 13 2 2005 293 303
47. S. Lee, A. Potamianos, and S. Narayanan Acoustic of children's speech: developmental changes of temporal and spectral parameters The Journal of the Acoustical Society of America (JASA) 105 3 1999 1455 1468
48. Lee, C., Narayanan, S., Pieraccini, R., 2001. Recognition of negative emotions from the speech signal. In: Proc. ASRU, Madonna di Campiglio. no pagination.
49. Lee, C.M., Narayanan, S.S., Pieraccini, R., 2002. Combining acoustic and language information for emotion recognition. In: Proc. Interspeech, Denver, pp. 873-376.
50. J.B. Lovins Development of a stemming algorithm Mechanical Translation and Computational Linguistics 11 1968 22 31
51. Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R., 1999. Performance measures for information extraction. In: Proc. DARPA Broadcast News Workshop, Herndon, VA, USA.
52. McGilloway, S., Cowie, R., Doulas-Cowie, E., Gielen, S., Westerdijk, M., Stroeve, S., 2000. Approaching automatic recognition of emotion from voice: a rough benchmark. In: Proc. ISCA workshop on Speech and Emotion, Newcastle, pp. 207-212.
53. Meyer, D., Leisch, F., Hornik, K., 2002. Benchmarking Support Vector Machines. Report Series No. 78, Adaptive Informations Systems and Management in Economics and Management Science, 19 pages.
54. D. Morrison, R. Wang, W. Xu, and L.C.D. Silva Incremental learning for spoken affect classification and its application in call-centres International Journal of Intelligent Systems Technologies and Applications 2 2007 242 254
55. Mozziconacci, S., 1998. Speech Variability and Emotion: Production and Perception. Ph.D. Thesis, Technical University Eindhoven.
56. Nogueiras, A., Moreno, A., Bonafonte, A., Mariño, J.B., 2001. Speech emotion recognition using hidden markov models. In: Proc. Eurospeech, Aalborg, pp. 2267-2270.
57. Nose, T., Kato, Y., Kobayashi, T., 2007. Style estimation of speech based on multiple regression hidden semi-markov model. In: Proc. Interspeech, Antwerp, pp. 2285-2288.
58. T. Nwe, S. Foo, and L.D. Silva Speech emotion recognition using hidden markov models Speech Communication 41 2003 603 623
59. A. Ortony, G.L. Clore, and A. Collins The Cognitive Structure of Emotions 1988 Cambridge University Press Cambridge, NY
60. P.-Y. Oudeyer The production and recognition of emotions in speech: features and algorithms International Journal of Human-Computer Studies 59 1-2 2003 157 183
61. T.V. Pernegger What's wrong with Bonferroni adjustment British Medical Journal 316 1998 1236 1238
62. R. Picard Affective Computing 1997 MIT Press Cambridge, MA
63. P. Pudil, J. Novovicova, and J. Kittler Floating search methods in feature selection Pattern Recognition Letters 15 1994 1119 1125
64. W. Rozeboom The fallacy of the null-hypothesis significance test Psychological Bulletin 57 1960 416 428
65. S. Salzberg On comparing classifiers: Pitfalls to avoid and a recommended approach Data Mining and Knowledge Discovery 1 3 1997 317 328
66. K.R. Scherer Vocal affect expression: a review and a model for future research Psychological Bulletin 99 1986 143 165
67. K.R. Scherer Vocal communication of emotion: a review of research paradigms Speech Communication 40 2003 227 256
68. Schröder, M., Pirker, H., Lamolle, M., 2006. First Suggestions for an Emotion Annotation and Representation Language. In: Devillers, L., Martin, J.-C., Cowie, R., Douglas-Cowie, E., Batliner, A. (Eds.), Proc. a Satellite Workshop of LREC 2006 on Corpora for Research on Emotion and Affect, Genoa, pp. 88-92.
69. M. Schröder, L. Devillers, K. Karpouzis, J.-C. Martin, C. Pelachaud, C. Peter, H. Pirker, B. Schuller, J. Tao, and I. Wilson What should a generic emotion markup language be able to represent? A. Paiva, R. Prada, R.W. Picard, Affective Computing and Intelligent Interaction 2007 Springer Berlin-Heidelberg 440 451
70. Schuller, B., Rigoll, G., Lang, M., 2003. Hidden markov model-based speech emotion recognition. In: Proc. ICASSP, Hong Kong, pp. 1-4.
71. Schuller, B., Müller, R., Lang, M., Rigoll, G., 2005. Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensemble. In: Proc. Interspeech, Lisbon, pp. 805-808.
72. Schuller, B., Reiter, S., Rigoll, G., 2006. Evolutionary feature generation in speech emotion recognition. In: Proc. Int. Conf. on Multimedia and Expo ICME 2006, Toronto, Canada, pp. 5-8.
73. Schuller, B., Batliner, A., Seppi, D., Steidl, S., Vogt, T., Wagner, J., Devillers, L., Vidrascu, L., Amir, N., Kessous, L., Aharonson, V., 2007a. The relevance of feature type for the automatic classification of emotional user states: low level descriptors and functionals. In: Proc. Interspeech, Antwerp, pp. 2253-2256.
74. Schuller, B., Seppi, D., Batliner, A., Meier, A., Steidl, S., 2007b. Towards more reality in the recognition of emotional speech. In: Proc. ICASSP, Honolulu, pp. 941-944.
75. Schuller, B., Wimmer, M., Mösenlechner, L., Kern, C., Arsic, D., Rigoll, G., 2008. Brute-Forcing Hierarchical Functionals for Paralinguistics: A Waste of Feature Space? In: Proc. ICASSP, Las Vegas, pp. 4501-4504.
76. Seppi, D., Gerosa, M., Schuller, B., Batliner, A., Steidl, S., 2008. Detecting Problems in Spoken Child-Computer Interaction. In: Proc. 1st Workshop on Child, Computer and Interaction, Chania, Greece.
77. M. Shami, and W. Verhelst Automatic classification of expressiveness in speech: a multi-corpus study Speaker Classification II C. Müller, Lecture Notes in Computer Science/Artificial Intelligence vol. 4441 2007 Springer Heidelberg - Berlin - New York 43 56
78. Steidl, S., 2009. Automatic Classification of Emotion-Related User States in Spontaneous Children's Speech. PhD Thesis, Logos Verlag, Berlin.
79. Steidl, S., Levit, M., Batliner, A., Nöth, E., Niemann, H., 2005. Of all things the measure is man: automatic classification of emotions and inter-labeler consistency. In: Proc. ICASSP, Philadelphia, pp. 317-320.
80. Steidl, S., Batliner, A., Nöth, E., Hornegger, A., 2008. Quantification of Segmentation and Fo Errors and Their Effect on Emotion Recognition. In: Text, Speech and Dialogue, 11th International Conference, TSD 2008, pp. 525-534.
81. Tato, R., Santos, R., Kompe, R., Pardo, J., 2002. Emotional space Improves Emotion Recognition. In: Proc. ICSLP 2002, pp. 2029-2032.
82. Vidrascu, L., Devillers, L., 2008. Anger detection performances based on prosodic and acoustic cues in several corpora. In: Devillers, L., Martin, J.-C., Cowie, R., Douglas-Cowie, E., Batliner, A. (Eds.), Proc. Workshop on Corpora for Research on Emotion and Affect at LREC 2008, Marrakech, pp. 23-27.
83. B. Vlasenko, B. Schuller, A. Wendemuth, and G. Rigoll Frame vs. turn-level: emotion recognition from speech considering static and dynamic processing A. Paiva, R. Prada, R.W. Picard, Affective Computing and Intelligent Interaction 2007 Springer Berlin-Heidelberg 139 147
84. Vogt, T., André, E., 2005. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proc. Multimedia and Expo (ICME05), Amsterdam, pp. 474-477.
85. J. Wagner, T. Vogt, and E. André A systematic comparison of different HMM designs for emotion recognition from acted and spontaneous speech A. Paiva, R. Prada, R.W. Picard, Affective Computing and Intelligent Interaction 2007 Springer Berlin-Heidelberg 114 125
86. I.H. Witten, and E. Frank Data Mining: Practical Machine Learning Tools and Techniques 2nd ed. 2005 Morgan Kaufmann San Francisco
87. You, M., Chen, C., Bu, J., Liu, J., Tao, J., 2006. Emotion recognition from noisy speech. In: Proc. ICME, Toronto, pp. 1653-1656.