LSTM-modeling of continuous emotions in an audiovisual affect recognition framework

Image and Vision Computing

Volumn 31, Issue 2, 2013, Pages 153-163

  Wöllmer, Martin ^a   Kaiser, Moritz ^a   Eyben, Florian ^a   Schuller, Björn ^a   Rigoll, Gerhard ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Context modeling; Emotion recognition; Facial movement features; Long short term memory

Indexed keywords

BRAIN;

AFFECTIVE COMPUTING; AUDIO-VISUAL AFFECT RECOGNITION; CONTEXT MODELING; DIMENSIONAL REPRESENTATION; EMOTION RECOGNITION; FACIAL MOVEMENTS; LONG SHORT-TERM MEMORY; RECOGNIZING HUMAN EMOTION;

HUMAN COMPUTER INTERACTION;

EID: 84886418479     PISSN: 02628856     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.imavis.2012.03.001     Document Type: Article

References (69)

1. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, J.G. Taylor, Emotion recognition in human-computer interaction, IEEE Signal Process. Mag. 18 (1) (2001) 32-80. (Pubitemid 32287669)
2. T. Ziemke, R. Lowe, On the role of emotion in embodied cognitive architectures: from organisms to robots, Cogn. Comput. 1 (1) (2009) 104-117.
3. L. Devillers, C. Vaudable, C. Chastagnol, Real-life emotion-related states detection in call centers: A cross-corpora study, Proc. of Interspeech, Makuhari, Japan, 2010, pp. 2350-2353.
4. M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. Maat, G. McKeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M. Valstar, M. Wöllmer, Building autonomous sensitive artificial listeners, IEEE Trans. Affective Comput. doi:10.1109/T-AFFC.2011.34.
5. B. Schuller, R. Müller, F. Eyben, J. Gast, B. Hörnler, M. Wöllmer, G. Rigoll, A. Höthker, H. Konosu, Being bored? Recognising natural interest by extensive audiovisual integration for real-life application, Image and Vision Computing Journal, Special Issue on Visual and Multimodal Analysis of Human Spontaneous Behavior, 27 (12), 2009, pp. 1760-1774.
6. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, M. Pantic, AVEC - the First International Audio/Visual Emotion Challenge, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 415-424.
7. B. Schuller, M.Wimmer, L.Mösenlechner, D. Arsic, G. Rigoll, Brute-forcingHierarchical Functionals for Paralinguistics: A Waste of Feature Space? Proc. of ICASSP, Las Vegas, NV, 2008, pp. 4501-4504.
8. S. Steidl, B. Schuller, A. Batliner, D. Seppi, The Hinterland of Emotions: Facing the Open-microphone Challenge, Proc. of ACII, Amsterdam, The Netherlands, 2009, pp. 690-697.
9. H. Gunes, B. Schuller, M. Pantic, R. Cowie, Emotion Representation, Analysis and Synthesis in Continuous Space: A Survey, Proc. of IEEE Conference on Face and Gesture Recognition, Santa Barbara, CA, USA, 2011, pp. 827-834.
10. M. Grimm, K. Kroschel, S. Narayanan, Support Vector Regression for Automatic Recognition of Spontaneous Emotions in Speech, Proc. of ICASSP, Honolulu, Hawaii, 2007, pp. 1085-1088.
11. F. Eyben, M. Wöllmer, A. Graves, B. Schuller, E. Douglas-Cowie, R. Cowie, On-line Emotion Recognition in a 3-D Activation-Valence-Time Continuum Using Acoustic and Linguistic Cues, Journal on Multimodal User Interfaces (JMUI), Special Issue on Real-time Affect Analysis and Interpretation: Closing the Loop in Virtual Agents, 3, 2009, pp. 7-19.
12. B. Schuller, B. Vlasenko, F. Eyben, G. Rigoll, A.Wendemuth, Acoustic Emotion Recognition: A Benchmark Comparison of Performances, Proc. of ASRU, Merano, Italy, 2009, pp. 552-557.
13. M. Wöllmer, A. Metallinou, F. Eyben, B. Schuller, S. Narayanan, Context-sensitive Multimodal Emotion Recognition from Speech and Facial Expression Using Bidirectional LSTM Modeling, Proc. of Interspeech, Makuhari, Japan, 2010, pp. 2362-2365.
14. G. McKeown, M.F. Valstar, M. Pantic, R. Cowie, The SEMAINE Corpus of Emotionally Coloured Character Interactions, Proc. of ICME, 2010, pp. 1-6.
15. B. Schuller, G. Rigoll, M. Lang, Hidden Markov Model-based Speech Emotion Recognition, Proc. of ICASSP, Hong Kong, China, 2003, pp. 1-4.
16. L.F. Barrett, E.A. Kensinger, Context is routinely encoded during emotion perception, Psychol. Sci. 21 (2010) 595-599.
17. M. Wöllmer, B. Schuller, F. Eyben, G. Rigoll, Combining Long Short-Term Memory and Dynamic Bayesian Networks for incremental emotion-sensitive artificial listening, IEEE J. Sel. Top. Sign. Proces. 4 (5) (2010) 867-881.
18. S. Hochreiter, J. Schmidhuber, Long Short-Term Memory, Neural Comput. 9 (8) (1997) 1735-1780. (Pubitemid 127462305)
19. S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, Gradient flow in recurrent nets: The difficulty of learning long-term dependencies, in: C. Kremer, J.F. Kolen (Eds.), A Field Guide to Dynamical Recurrent Neural Networks, IEEE Press, 2001, pp. 1-15.
20. A. Graves, J. Schmidhuber, Framewise phoneme classification with bidirectional LSTM and other neural network architectures, Neural Networks 18 (5-6) (2005) 602-610. (Pubitemid 43186580)
21. S. Fernandez, A. Graves, J. Schmidhuber, An Application of Recurrent Neural Networks to Discriminative Keyword Spotting, Proc. of ICANN, Porto, Portugal, 2007, pp. 220-229.
22. M. Wöllmer, F. Eyben, J. Keshet, A. Graves, B. Schuller, G. Rigoll, Robust Discriminative Keyword Spotting for Emotionally Colored Spontaneous Speech Using Bidirectional LSTM Networks, Proc. of ICASSP, Taipei, Taiwan, 2009, pp. 3949-3952.
23. M. Wöllmer, F. Eyben, A. Graves, B. Schuller, G. Rigoll, Bidirectional LSTM networks for context-sensitive keyword detection in a cognitive virtual agent framework, Cogn. Comput. 2 (3) (2010) 180-190.
24. M. Wöllmer, F. Eyben, B. Schuller, G. Rigoll, A Multi-stream ASR Framework for BLSTM Modeling of Conversational Speech, Proc. of ICASSP, Prague, Czech Republic, 2011, pp. 4860-4863.
25. M. Wöllmer, B. Schuller, G. Rigoll, A Novel Bottleneck-BLSTM Front-end for Feature-level Context Modeling in Conversational Speech Recognition, Proc. of ASRU, Waikoloa, Big Island, Hawaii, 2011, pp. 36-41.
26. M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, R. Cowie, Abandoning Emotion Classes - Towards Continuous Emotion Recognition with Modelling of Long-range Dependencies, Proc. of Interspeech, Brisbane, Australia, 2008, pp. 597-600.
27. M.A. Nicolaou, H. Gunes, M. Pantic, Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space, IEEE Trans. Affect. Comput. 2 (2011) 92-105.
28. F. Eyben, M. Wöllmer, B. Schuller, open SMILE - The Munich Versatile and Fast Open-source Audio Feature Extractor, Proc. of ACM Multimedia, Firenze, Italy, 2010, pp. 1459-1462.
29. D. Arsic, B. Hörnler, B. Schuller, G. Rigoll, A Hierarchical Approach for Visual Suspicious Behavior Detection in Aircrafts, Proceedings of the 16th international conference on Digital Signal Processing, 2009, pp. 639-645.
30. A. Sayedelahl, P. Fewzee, M. Kamel, F. Karray, Audio-based Emotion Recognition from Natural Conversations Based on Co-Occurrence Matrix and Frequency Domain Energy Distribution Features, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 407-414.
31. L. Cen, Z.L. Yu, M.H. Dong, Speech Emotion Recognition System based on L1 Regularized Linear Regression and Decision Fusion, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 332-340.
32. S. Pan, J. Tao, Y. Li, The CASIA Audio Emotion Recognition Method for Audio/Visual Emotion Challenge 2011, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 388-395.
33. G. Ramirez, T. Baltrusaitis, L.P. Morency, Modeling Latent Discriminative Dynamic of Multi-dimensional Affective Signals, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 396-406.
34. J.C. Kim, H. Rao, M.A. Clements, Investigating the Use of Formant Based Features for Detection of Affective Dimensions in Speech, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 369-377.
35. M. Glodek, S. Tschechne, G. Layher, M. Schels, T. Brosch, S. Scherer, M. Kächele, M. Schmidt, H. Neumann, G. Palm, F. Schwenker, Multiple Classifier Systems for the Classification Of Audio-Visual Emotional States, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 359-368.
36. J.R.J. Fontaine, K.R. Scherer, E.B. Roesch, P. Ellsworth, The world of emotions is not two-dimensional, Psychol. Sci. 18 (2) (2007) 1050-1057.
37. B. Schuller, S. Steidl, A. Batliner, F. Schiel, J. Krajewski, The Interspeech 2011 Speaker State Challenge, Proc. of Interspeech 2011, Florence, Italy, 2011, pp. 3201-3204.
38. A. Lee, T. Kawahara, Recent Development of Open-source Speech Recognition Engine Julius, Proc. of APSIPA ASC, Sapporo, Japan, 2009, pp. 131-137.
39. A. Stupakov, E. Hanusa, D. Vijaywargi, D. Fox, J. Bilmes, The design and collection of COSINE, a multi-microphone in situ speech corpus recorded in noisy environments, Comput. Speech Lang. 26 (1) (2011) 52-66.
40. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. McRorie, J.C. Martin, L. Devillers, S. Abrilian, A. Batliner, N. Amir, K. Karpouzis, The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data, Affective Computing and Intelligent Interaction, vol. 4738/2007, Springer, 2007, pp. 488-500.
41. C. Shan, S. Gong, P.W. McOwan, Facial expression recognition based on local binary patterns: A comprehensive study, Image Vision Comput. 27 (6) (2009) 803-816.
42. I. Kotsia, I. Pitas, Facial expression recognition in image sequences using geometric deformation features and support vector machines, IEEE Trans. Image Process. 16 (1) (2007) 172-187. (Pubitemid 46437480)
43. P. Yang, Q. Liu, D.N. Metaxas, Boosting encoded dynamic features for facial expression recognition, Pattern Recognit. Lett. 30 (2) (2009) 132-139.
44. I. Kotsia, I. Buciu, I. Pitas, An analysis of facial expression recognition under partial facial image occlusion, Image Vision Comput 26 (7) (2008) 1052-1067.
45. S. Kumano, K. Otsuka, J. Yamato, E. Maeda, Y. Sato, Pose-invariant facial expression recognition using variable-intensity templates, Int. J. Comput. Vis. 83 (2) (2009) 178-194.
46. Y. Tian, T. Kanade, J.F. Cohn, Handbook of Face Recognition, Facial Expression Analysis, Springer, London, 2011, pp. 487-519.
47. F. Eyben, M. Wöllmer, M. Valstar, H. Gunes, B. Schuller, M. Pantic, String-based audiovisual fusion of behavioural events for the assessment of dimensional affect, Proc. of FG, 2011, pp. 322-329.
48. M. Wöllmer, M. Al-Hames, F. Eyben, B. Schuller, G. Rigoll, A multidimensional dynamic time warping algorithm for efficient multimodal fusion of asynchronous data streams, Neurocomputing 73 (1-3) (2009) 366-380.
49. A. Metallinou, S. Lee, S. Narayanan, Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice, International Symposium on Multimedia, Los Alamitos, CA, USA, 2008, pp. 250-257.
50. Z. Zeng, J. Tu, B. Pianfetti, T.S. Huang, Audio-visual affective expression recognition through multistream fused HMM, IEEE Trans. Multimedia 10 (4) (2008) 570-577. (Pubitemid 351711233)
51. O. Collignon, S. Girard, F. Gosselin, S. Roy, D. Saint-Amour, M. Lassonde, F. Lepore, Audio-visual integration of emotion expression, Brain Res. 1242 (2008) 126-135.
52. A. Cruz, B. Bhanu, S. Yang, A psychologically-inspired match-score fusion model for video-based facial expression recognition, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 341-350.
53. M. Dahmane, J. Meunier, Continuous emotion recognition using Gabor energy filters, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 351-358.
54. P.A. Viola, M.J. Jones, Robust real-time face detection, Int. J. Comput. Vis. 57 (2) (2004) 137-154.
55. T. Ojala, M. Pietikäinen, T. Mäenpää, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Trans. Pattern Anal. Mach. Intell. 24 (7) (2002) 971-987. (Pubitemid 34835471)
56. D. Greig, Video object detection speedup using staggered sampling, IEEE Workshop on Applications of Computer Vision (WACV), 2009, pp. 23-29.
57. G.R. Bradski, Computer vision face tracking for use in a perceptual user interface, Tech. Rep. Q2, Intel Technol. J. (1998) 1-15.
58. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I.H. Witten, The WEKA data mining software: An update, ACM SIGKDD Explor. Newslett. 11 (2009) 10-18.
59. H. Meng, N. Bianchi-Berthouze, Naturalistic affective expression classification by a multi-stage approach based on Hidden Markov Models, Proc. of First International Audio/Visual Emotion Challenge and Workshop (AVEC 2011) held in conjunction with ACII, Memphis, Tennessee, USA, 2011, pp. 378-387.
60. M. Schuster, K.K. Paliwal, Bidirectional recurrent neural networks, IEEE Trans. Signal Process. 45 (1997) 2673-2681. (Pubitemid 127766336)
61. F. Gers, J. Schmidhuber, F. Cummins, Learning to forget: continual prediction with LSTM, Neural Comput. 12 (10) (2000) 2451-2471.
62. A. Graves, Supervised sequence labelling with recurrent neural networks, Ph.D. thesis, Technische Universität München (2008).
63. M.Wöllmer, B. Schuller, A. Batliner, S. Steidl, D. Seppi, Tandemdecoding of children's speech for keyword detection in a child-robot interaction scenario, IEEE Trans. Audio Speech Lang. Process. 7 (4) (2011) 1-26.
64. M. Wöllmer, C. Blaschke, T. Schindl, B. Schuller, B. Färber, S. Mayer, B. Trefflich, On-line driver distraction detection using Long Short-Term Memory, IEEE Trans. Intell. Transp. Syst. 12 (2) (2011) 574-582.
65. M. Wöllmer, A. Metallinou, N. Katsamanis, B. Schuller, S. Narayanan, Analyzing the Memory of BLSTM Neural Networks for Enhanced Emotion Classification in Dyadic Spoken Interactions, Proc. of ICASSP, Kyoto, Japan, 2012.
66. I.H. Witten, E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd Edition Morgan Kaufmann, San Francisco, 2005.
67. G.W. Snedecor, W.G. Cochran, Statistical methods, 8th ed. Iowa State University Press, 1989.
68. F. Grezl, M. Karafiat, K. Stanislav, J. Cernocky, Probabilistic and Bottle-neck Features for LVCSR of Meetings, Proc. of ICASSP, Honolulu, Hawaii, 2007, pp. 757-760.
69. S. Bengio, An asynchronous Hidden Markov Model for audio-visual speech recognition, Adv. NIPS 15 (2003) 1-8.