A multitask approach to continuous five-dimensional affect sensing in natural speech

ACM Transactions on Interactive Intelligent Systems

Volumn 2, Issue 1, 2012, Pages 1-29

  Eyben, Florian ^a   Wöllmer, Martin ^a   Schuller, Björn ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Audio features; Dimensional affect; Emotion recognition; Long short term memory; Neural networks; SEMAINE

Indexed keywords

BRAIN; CHEMICAL ACTIVATION; INTERACTIVE COMPUTER SYSTEMS; LEARNING SYSTEMS; LONG SHORT-TERM MEMORY; NEURAL NETWORKS; REAL TIME SYSTEMS; SUPPORT VECTOR REGRESSION;

AFFECT RECOGNITION; AFFECTIVE STATE; AUDIO FEATURES; CORRELATION COEFFICIENT; DIMENSIONAL AFFECT; EMOTION RECOGNITION; SEMAINE; TECHNICAL SYSTEMS;

FEEDFORWARD NEURAL NETWORKS;

EID: 84983561287     PISSN: 21606455     EISSN: 21606463     Source Type: Journal    
DOI: 10.1145/2133366.2133372     Document Type: Article

References (73)

1. BATLINER, A., SEPPI, D., STEIDL, S., AND SCHULLER, B. 2010. Segmenting into adequate units for automatic recognition of emotion-relatedepisodes: A speech-based approach.Advan. Hum. Comput. Interact. article ID 782802..
2. BATLINER, A., STEIDL, S., SCHULLER, B., SEPPI, D., VOGT, T., WAGNER, J., DEVILLERS, L., VIDRASCU, L., KESSOUS, V. A. L, AND AMIR, N. 2011. Whodunnit\-Searching for the most important feature types signalling emotion-related user states in speech. Comput. Speech Lang. 25, 1..
3. BURKHARDT, F., PAESCHKE, A., ROLFES, M., SENDLMEIER, W., AND WEISS, B. 2005.A database of german emotional speech. In Proceedings of the Interspeech Conference. 1517-1520..
4. BUSSO, C., LEE, S., AND NARAYANAN, S. S. 2007. Using neutral speech models for emotional speech analysis. In Proceedings of the Interspeech Conference. 2225-2228..
5. CARIDAKIS, G., MALATESTA, L., KESSOUS, L., AMIR, N., RAOUZAIOU, A., AND KARPOUZIS, K. 2006. Modelingnaturalistic affective states via facial and vocal expressions recognition. In Proceedings of the ACM International Conference on Multimodal Interfaces. 146-154..
6. CHANG, C.-C. AND LIN, C.-J. 2001.LibSVM:A Library for Support Vector Machines.http://www.csie.ntu.edu.tw/cjlin/libsvm..
7. COHEN, J., COHEN, P., WEST, S. G., AND AIKEN, L. S. 2003.Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, 2nd ed. Lawrence Erlbaum Associates, Hillsdale, NJ..
8. COWIE, R., DOUGLAS-COWIE, E., SAVVIDOU, S., MCMAHON, E., SAWEY, M., AND SCHRÖDER, M. 2000. Feeltrace: an instrument for recording perceived emotion in real time. In Proceedings of the ISCA Workshop on Speech and Emotion. 19-24..
9. DEVILLERS, L., VIDRASCU, L., AND LAMEL, L. 2005. Challenges in real-life emotion annotation and machine learning based detection. Neural Netw. 18, 4, 407-422..
10. DOUGLAS-COWIE, E., COWIE, R., SNEDDON, I., COX, C., LOWRY, O., MCRORIE, M., MARTIN, J. C., DEVILLERS, L., ABRILIAN, S., BATLINER, A., AMIR, N., AND KARPOUZIS, K. 2007a. The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data. In Affective Computing and IntelligentInteraction. Springer, 488-500..
11. DOUGLAS-COWIE, E., COWIE, R., SNEDDON, I., COX, C., O., L., MCRORIE, M., MARTIN, J., DEVILLERS, L., ABRILIAN, S., BATLINER, A., AMIR, N., AND KARPOUZIS, K. 2007b. The humaine database: Addressing the collection and annotation of naturalistic and induced emotional data. In Lecture Notes in Computer Science,Springer, vol. 4738, 488-501..
12. EKMAN, P. AND FRIESEN, W. V. 1975. Unmasking the Face: A Guide to Recognizing Emotions from Facial Expressions. Prentice Hall, Englewood Cliffs, NJ..
13. EYBEN, F., WÖLLMER, M., GRAVES, A., SCHULLER, B., DOUGLAS-COWIE, E., AND COWIE, R. 2010a. On-line emotion recognition ina3-d activation-valence-time continuum using acoustic and linguistic cues. J. Multimodal User Interfaces 3, 1-2, 7-19..
14. EYBEN, F., WÖLLMER, M., AND SCHULLER, B. 2010b. openSMILE\-The Munich versatile and fast open-source audio feature extractor In Proceedings of ACM Multimedia Conference. 1459-1462..
15. EYBEN, F., WÖLLMER, M., VALSTER, M., GUNES, H., SCHULLER, B., AND PANTIC, M. 2011. String-based audiovisual fusion of behavioural events for the assessment of dimensional affect. In Proceedings FG Conference (to appear)..
16. FERNANDEZ, S., GRAVES, A., AND SCHMIDHUBER, J. 2008. Phoneme recognition in timit with blstm-ctc. Tech. rep., IDSIA..
17. FONTAINE, J. R. J., SCHERER, K. R., ROESCH, E. B., AND ELLSWORTH, P. C. 2007. The world of emotions is not two-dimensional. Psychol. Sci. 18, 2, 1050-1057..
18. FRAGOPANAGOS, N. AND TAYLOR, J. G. 2005. Emotion recognition in human-computer interaction. Neural Netw. 18, 4, 389-405..
19. GLOWINSKI, D., CAMURRI, A., VOLPE, G., DAEL, N., AND SCHERER, K. 2008. Technique for automatic emotion recog-nition by body gesture analysis. In Proceedings ofComputer Vision and Pattern Recognition Workshops. 1-6..
20. GRAVES, A., FERNANDEZ, S., AND SCHMIDHUBER, J. 2005. Bidirectional lstm networks for improved phoneme classification and recognition. In Proceedings ofICANN. Vol. 18. 602-610..
21. GRAVES, A. AND SCHMIDHUBER, J. 2005. Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18, 5-6, 602-610..
22. GRIMM, M., KROSCHEL, K., AND NARAYANAN, S. 2007a. Support vector regression for automatic recognition of spontaneous emotions in speech. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP). Vol. 4. IEEE..
23. GRIMM, M., MOWER, E., KROSCHEL, K., AND NARAYANAN, S. 2007b. Primitives based estimation and evaluation of emotions in speech. Speech Comm. 49, 787-800..
24. GUNES, H. AND PANTIC, M. 2010a. Automatic, dimensional and continuous emotion recognition. Int. J. Synth. Emot. 1, 1, 68-99..
25. GUNES, H. AND PANTIC, M. 2010b. Automatic measurement of affect in dimensional and continuous spaces: Why, what, andhow? In Proceedings of the Conference on MeasuringBehavior. 122-126..
26. GUNES, H. AND PANTIC, M. 2010c. Dimensional emotion prediction from spontaneous head gestures for inter-actionwith sensitive artificial listeners. In Proceedings of International Conference on Intelligent Virtual Agents. 371-377..
27. HALL, M. A. 1998. Correlation-based feature subset selection for machine learning. Ph.D. thesis, University of Waikato, Hamilton, New Zealand..
28. HOCHREITER, S., BENGIO, Y., FRASCONI, P., AND SCHMIDHUBER, J. 2001. Gradient flow in recurrent nets: the difficulty oflearning long-term dependencies. In A Field Guide to Dynamical Recurrent Neural Networks, S. C. Kremer and J. F Kolen, Eds., IEEE Press..
29. HOCHREITER, S. AND SCHMIDHUBER, J. 1997. Long short-term memory. Neural Comput. 9, 8, 1735-1780..
30. IOANNOU, S., RAOUZAIOU, A., TZOUVARAS, V., MAILIS, T., KARPOUZIS, K., AND KOLLIAS, S. 2005. Emotion recognition through facial expression analysis based on a neurofuzzy method. J. Neural Netw. 18, 423-435..
31. LEE, C., BUSSO, C., LEE, S., AND NARAYANAN, S. 2009. Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions. In Proceedings of the Interspeech Conference. 1983-1986..
32. LICHTENAUER, J., SHEN, J., VALSTAR, M., AND PANTIC, M. 2010. Cost-Effective solution to synchronised audio-visual data capture using multiple sensors. J. Vis. Comm. Image Represent., 1-39..
33. MCKEOWN, G., VALSTAR, M. F., PANTIC, M., AND COWIE, R. 2010. The semaine corpus of emotionally coloured character interactions. In Proceedings of the ICME Conference. IEEE, 1-6..
34. METALLINOU, A., KATSAMANIS, A., WANG, Y., AND NARAYANAN, S. S. 2011. Tracking changes in continuous emotion states using body language and prosodic cues. In Proceedings of the International Conference onAcoustics, Speech and Signal Processing. IEEE, 2288-2291..
35. MOWER, E., MATARIC, M. J., ANDNARAYANAN, S. S. 2011. Aframe work for automatic human emotion classification using emotional profiles. IEEE Trans. Audio, Speech Lang. Process. 19, 5, 1057-1070..
36. MOWER, E. AND NARAYANAN, S. S. 2011. A hierarchical static-dynamic framework for emotion classification. In Proceedings of International Conference onAcoustics, Speech and Signal Processing. IEEE, 2372-2375..
37. NICOLAOU, M., GUNES, H., AND PANTIC, M. 2010. Audio-visual classification and fusion of spontaneous affective data in likelihood space. In Proceedings ofIEEE International Conference on Pattern Recognition. 3695-3699..
38. OUDEYER, P. Y. 2003. The production and recognition of emotions in speech: Features and algorithms. Int. J. Hum.-Comput. Studies 59, 157-183..
39. PÉREZ-ORTIZ, J. A., GERS, F A., ECK, D., AND SCHMIDHUBER, J. 2003. Kalman filters improve lstm network performance in problems unsolvable by traditional recurrent nets. Neural Netw. 16, 2, 241-250..
40. PETERS, C. AND O'SULLIVAN, C. 2002. Synthetic vision and memory for autonomous virtual humans. Comput. Graph. Forum 21, 4, 743-753..
41. RIEDMILLER, M. AND BRAUN, H. 1993.A direct adaptive method for faster backpropagation learning: The rprop algorithm. In Proceedings of the IEEE International Conference on Neural Networks. 586-591..
42. SCHMIDT, E. M. AND KIM, Y. E. 2010. Prediction oftime-varying musical mood distributions from audio. In Proceedings of the International Society for Music Information Retrieval Conference..
43. SCHULLER, B., BATLINER, A., STEIDL, S., AND SEPPI, D. 2010a. Recognising realistic emotions and affect in speech: State of the art and lessons learnt from the first challenge. Speech Comm. (Special Issue on Sensing Emotion and AffectFacing Realism in Speech Processing) (to appear)..
44. SCHULLER, B., MÜLLER, R., EYBEN, F., GAST, J., HÖRNLER, B., WÖLLMER, M., RIGOLL, G., HÖTHKER, A., AND KONOSU, H. 2009a. Being bored? Recognising natural interest by extensive audiovisual integration for real-life application. Image Vis. Comput. J. 27, 12, 1760-1774..
45. SCHULLER, B., REITER, S., AND RIGOLL, G. 2006. Evolutionary feature generation in speech emotion recognition. In Proceedings of the International Conference on Multimedia and Expo (ICME). 5-8..
46. SCHULLER, B. AND RIGOLL, G.2006. Timing levels in segment-based speech emotion recognition. In Proceedings of the Inter speech Conference. 1818-1821..
47. SCHULLER, B., RIGOLL, G., AND LANG, M. 2003. Hidden markov model-based speech emotion recognition. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing. Vol. II. IEEE, 1-4..
48. SCHULLER, B., SEPPI, D., BATLINER, A., MAIER, A., AND STEIDL, S. 2007a. Towards more reality in the recognition of emotional speech. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing. Vol. IV. IEEE, 941-944..
49. SCHULLER, B., STEIDL, S., AND BATLINER, A. 2009b. The INTERSPEECH 2009 emotion challenge. In Proceedings of theInterspeech Conference. 312-315..
50. SCHULLER, B., STEIDL, S., BATLINER, A., BURKHARDT, F., DEVILLERS, L., MÜLLER, C., AND NARAYANAN, S. 2010b. The interspeech 2010 paralinguistic challenge. In Proceedings of the Interspeech Conference. 2794-2797..
51. SCHULLER, B., STEIDL, S., BATLINER, A., SCHIEL, F., AND KRAJEWSKI, J. 2011. The interspeech 2011 speaker state challenge. In Proceedings of the Interspeech Conference..
52. SCHULLER, B., VLASENKO, B., EYBEN, F., RIGOLL, G., AND WENDEMUTH, A. 2009c. Acoustic emotion recognition: A benchmark comparison ofperformances. In Proceedings of theASRUConference. IEEE..
53. SCHULLER, B., VLASENKO, B., EYBEN, F., WÖLLMER, M., STUHLSATZ, A., WENDEMUTH, A., AND RIGOLL, G. 2010c. Cross-corpus acoustic emotion recognition: Variances and strategies. IEEE Trans. Affective Comput. 1, 2..
54. SCHULLER, B., VLASENKO, B., MINGUEZ, R., RIGOLL, G., AND WENDEMUTH, A. 2007b. Comparing one and two-stage acoustic modeling in the recognition of emotion in speech. In Proceedings of the Automatic Speech Recognition and Understanding Workshop. 596-600..
55. SCHULLER, B., WIMMER, M., MÖSENLECHNER, L., KERN, C., ARSIC, D., AND RIGOLL, G. 2008. Brute-Forcing hier-archical functionals for paralinguistics: A waste of feature space? In Proceedings of the International Conference onAcoustics, Speech and Signal Processing. 4501-4504..
56. SCHULLER, B., ZACCARELLI, R., ROLLET, N., AND DEVILLERS, L. 2010d. Cinemo a french spoken language resource for complex emotions: Facts and baselines. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC)..
57. SCHUSTER, M. AND PALIWAL, K. K. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45, 2673-2681..
58. STEIDL, S. 2009.Automatic Classification of Emotion-Related User States in Spontaneous Children's Speech. Logos, Berlin..
59. STEIDL, S., SCHULLER, B., BATLINER, A., AND SEPPI, D. 2009. The hinterland of emotions: Facing the open-microphone challenge. In Proceedings of the 4th International HUMAINE Association Conference on Affective Computing and Intelligent Interaction (ACII). Vol. I. IEEE, 690-697..
60. STREIT, M., BATLINER, A., AND PORTELE, T. 2006. Emotions analysis and emotion-handling subdia-logues. In SmartKom: Foundations of Multimodal Dialogue Systems, W. Wahlster, Ed., Springer, 317-332..
61. VLASENKO, B., SCHULLER, B., WENDEMUTH, A., AND RIGOLL, G. 2007. Frame vs. turn-level: Emotion recognition from speech considering static and dynamic processing. In Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction (ACII), Lecture Notes in Computer Science, vol. 4738. Springer, 139-147..
62. VLASENKO, B. AND WENDEMUTH, A. 2007. Tuning hidden markov model for speech emotion recognition. In Proceedings of DAGA 33rd German Annual Conference on Acoustics..
63. VOGT, T. AND ANDRE, E. 2005. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In Proceedings of the ICME Conference. 474-477..
64. WERBOS, P. 1990. Backpropagation through time: What it does and how to do it. Proc. IEEE 78, 1550-1560..
65. WITTEN, I. H. AND FRANK, E. 2005. Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. Morgan Kaufmann, San Francisco..
66. WÖLLMER, M., EYBEN, F., REITER, S., SCHULLER, B., COX, C., DOUGLAS-COWIE, E., AND COWIE, R. 2008. Abandoning emotion classes - Towards continuous emotion recognition with modelling of long-range dependencies. In Proceedings of the Interspeech Conference. 597-600..
67. WÖLLMER, M., EYBEN, F., SCHULLER, B., DOUGLAS-COWIE, E., AND COWIE, R. 2009. Data-driven clustering in emotional space for affect recognition using discriminatively trained LSTM networks. In Proceedings of Interspeech Conference. 1595-1598..
68. WÖLLMER, M., METALLINOU, A., EYBEN, F., SCHULLER, B., AND NARAYANAN, S. 2010a. Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. In Proceedings of Inter speech Conference. 2362-2365..
69. WÖLLMER, M., SCHULLER, B., EYBEN, F., AND RIGOLL, G. 2010b. Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening. IEEE J. Select. Topics Signal Process. 4, 5, 867-881..
70. WU, D., PARSONS, T., MOWER, E., AND NARAYANAN, S. S. 2010a. Speech emotion estimation in 3d space. In Proceedings of the ICME Conference. 737-742..
71. WU, D., PARSONS, T., AND NARAYANAN, S. S. 2010b. Acoustic feature analysis in speech emotion primitives estimation. In Proceedings of the Interspeech Conference. 785-788..
72. YEE, P. V. AND HAYKIN, S. 2001. Regularized Radial Basis Function Networks: Theory andApplications. John Wiley..
73. ZENG, Z., PANTIC, M., ROISMAN, G. I., AND HUANG, T. 2009. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 1, 39-58..