Context-sensitive learning for enhanced audiovisual emotion classification (Extended abstract)

2015 International Conference on Affective Computing and Intelligent Interaction, ACII 2015

Volumn , Issue , 2015, Pages 463-469

  Metallinou, Angeliki ^a   Katsamanis, Athanasios ^b   Wollmer, Martin ^c   Eyben, Florian ^c   Schuller, Bjorn ^c   Narayanan, Shrikanth ^d  

^a AMAZON   (United States)

^b NATIONAL TECHNICAL UNIVERSITY OF ATHENS   (Greece)

^c AUDEERING GMBH   (Germany)

^d UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

context modeling; emotion recognition; long short term memory

Indexed keywords

BRAIN; HIDDEN MARKOV MODELS; INTELLIGENT COMPUTING; MARKOV PROCESSES; SPEECH RECOGNITION;

CONTEXT MODELING; CONTEXT-SENSITIVE LEARNING; EMOTION CLASSIFICATION; EMOTION RECOGNITION; EMOTIONAL EXPRESSIONS; EMOTIONAL INTERACTIONS; HIERARCHICAL HIDDEN MARKOV MODEL; LONG SHORT TERM MEMORY;

BEHAVIORAL RESEARCH;

EID: 84964055236     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ACII.2015.7344611     Document Type: Conference Paper

References (36)

1. R. El Kaliouby, P. Robinson, and S. Keates, "Temporal context and the recognition of emotion from facial expression," in HCI International, Crete, June 2003.
2. J. M. Carroll and J. A. Russell, "Do facial expressions signal specific emotions? judging emotion from the face in context," Journal of Personality and Social Psychology, vol. 70, pp. 205-218, 1996.
3. H. R. Knudsen and L. H. Muzekari, "The effects of verbal statements of context on facial expressions of emotion," Journal of Nonverbal Behavior, vol. 7, pp. 202-212, 1983.
4. T. Masuda, P.C. Ellsworth, B. Mesquita, J. Leu, S. Tanida, and E. Van de Veerdonk, "Placing the face in context: Cultural differences in the perception of facial emotion," Journal of Personality and Social Psychology, vol. 94, pp. 365-381, 2008.
5. A. Mehrabian, "Communication without words," Psychology today, vol. 2, pp. 53-56, 1968.
6. B. de Gelder and J. Vroomen, "The perception of emotions by ear and by eye," Cognition and Emotion, pp. 289-311, May 2000.
7. K. Oatley and J. M. Jenkins, Understanding Emotions, Blackwell Publishers Ltd, 1996.
8. M. Wollmer, A. Metallinou, F. Eyben, B. Schuller, and S. Narayanan, "Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling," in Proc. of Interspeech, Japan, 2010.
9. S. Fine, Y. Singer, and N. Tishby, "The Hierarchical Hidden Markov Model: Analysis and applications," Machine Learning, vol. 32, pp. 41-62, 1998.
10. A. Graves, S. Fernandez, M. Liwicki, H. Bunke, and J. Schmidhuber, "Unconstrained online handwriting recognition with recurrent neural networks," Advances in Neural Information Processing Systems, vol. 20, pp. 1-8, 2008.
11. A. McCabe and J. Trevathan, "Handwritten signature verifcation using complementary statistical models," Journal Of Computers, vol. 4, pp. 670-680, 2009.
12. F. Weninger, J. Geiger, M. Wollmer, B. Schuller, and G. Rigoll, "Feature Enhancement by Deep LSTM Networks for ASR in Reverberant Multisource Environments," Computer Speech and Language, vol. 28, no. 4, pp. 888-902, July 2014.
13. E. Marchi, F. Vesperini, F. Weninger, F. Eyben, S. Squartini, and B. Schuller, "Non-Linear Prediction with LSTM Recurrent Neural Networks for Acoustic Novelty Detection," in Proceedings 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland, July 2015, IEEE, to appear.
14. R. Bruckner and B. Schuller, "Social Signal Classification Using Deep BLSTM Recurrent Neural Networks," in Proceedings 39th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2014, Florence, Italy, May 2014, pp. 4856-4860, IEEE.
15. J. Geiger, E. Marchi, F. Weninger, B. Schuller, and G. Rigoll, "The TUM system for the REVERB Challenge: Recognition of Reverberated Speech using Multi-Channel Correlation Shaping Dereverberation and BLSTM Recurrent Neural Networks," in Proceedings REVERB Workshop, held in conjunction with ICASSP 2014 and HSCMA 2014, Florence, Italy, May 2014, IEEE, pp. 1-8, IEEE.
16. E. Coutinho, F. Weninger, K. Scherer, and B. Schuller, "The Munich LSTM-RNN Approach to the MediaEval 2014 "Emotion in Music" Task," in Proceedings of the MediaEval 2014 Multimedia Benchmark Workshop, Martha Larson, Bogdan Ionescu, Xavier Anguera, Maria Eskevich, Pavel Korshunov, Markus Schedl, Mohammad Soleymani, Georgios Petkos, Richard Sutcliffe, Jaeyoung Choi, and Gareth J.F. Jones, Eds., Barcelona, Spain, October 2014, CEUR.
17. M.J.F. Gales and S.J. Young, "The application of Hidden Markov Models in speech recognition," Foundations and Trends in Signal Processing, vol. 1, pp. 195-304, 2008.
18. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book, Entropic Cambridge Research Laboratory, Cambridge, England, 2006.
19. A. Graves, S. Fernandez, and J. Schmidhuber, "Bidirectional LSTM networks for improved phoneme classification and recognition," in Proc. of ICANN, Poland, 2005, vol. 18, pp. 602-610.
20. C. Busso, M. Bulut, C-C Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan, "IEMOCAP: interactive emotional dyadic motion capture database," Language Resources and Evaluation, vol. 42, pp. 335-359, 2008.
21. M. Wollmer, F. Eyben, B. Schuller, E. Douglas-Cowie, and R. Cowie, "Data-driven clustering in emotional space for affect recognition using discriminatively trained LSTM networks," in Proc. of Interspeech, UK, 2009, pp. 1595-1598.
22. I. Cohen, A. Garg, and T. S. Huang, "Emotion recognition from facial expressions using multilevel HMM," Neural Information Processing Systems, 2000.
23. A. Metallinou, M. Wollmer, A. Katsamanis, F. Eyben, B. Suller, and S. Narayanan, "Context-sensitive learning for enhanced audiovisual emotion classification," IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 184-198, 2012.
24. C.-C. Lee, C. Busso, S. Lee, and S. Narayanan, "Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions," in Proc. of Interspeech, UK, 2009.
25. I. Cohen, Q. T. Xiang, S. Zhou, X. Sean, Z. Thomas, and T. S. Huang, "Feature selection using principal feature analysis," 2002.
26. A. Metallinou, C.Busso, S. Lee, and S. Narayanan, "Visual emotion recognition using compact facial representations and viseme information," in Proc. of ICASSP, Dallas, Texas, 2010, pp. 2474-2477.
27. P. Boersma, "Praat, a system for doing phonetics by computer.," Glot International, vol. 5, no. 9/10, pp. 341-345, 2001.
28. F. Eyben, M. Wollmer, and B. Schuller, "openSMILE\-The Munich versatile and fast open-source audio feature extractor," in Proc. of ACM Multimedia, Firenze, Italy, 2010.
29. M. A. Hall, Correlation-based Feature Selection for Machine Learning, Ph.D. thesis, University of Waikato, 1999.
30. Ian H. Witten and Eibe Frank, Data Mining: Practical machine learning tools and techniques, Morgan Kaufmann, San Francisco, 2nd edition edition, 2005.
31. A. V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, "A coupled HMM for audio-visual speech recognition," in Proc. of ICASSP, 2002, pp. 2013-2016.
32. M. Brand, N. Oliver, and A. Pentland, "Coupled Hidden Markov Models for complex action recognition," Proc. of CVPR, 1997.
33. G. Gravier, G. Potamianos, and C. Neti, "Asynchrony modeling for audio-visual speech recognition," in Proc. of the 2nd Intl. Conf. on Human Language Technology Research, San Francisco, CA, USA, 2002, HLT '02, pp. 1-6, Morgan Kaufmann Publishers Inc.
34. SPSS Inc., "SPSS base 10.0 for windows user's guide," SPSS Inc., Chicago IL., 1999.
35. J. A. Russell, J.-A. Bachorowski, and J.-M. Fernndez-Dols, "Facial and vocal expressions of emotion," Annual Review of Psychology, vol. 54, pp. 329-349, February 2003.
36. A. Metallinou, S. Lee, and S. Narayanan, "Decision level combination of multiple modalities for recognition and analysis of emotional expression," in Proc. of ICASSP, Dallas, Texas, 2010, pp. 2462-2465.