Emotion recognition in the wild: incorporating voice and lip activity in multimodal decision-level fusion

ICMI 2014 - Proceedings of the 2014 International Conference on Multimodal Interaction

Volumn , Issue , 2014, Pages 473-480

  Ringeval, Fabien ^a   Amiriparian, Shahin ^a   Eyben, Florian ^a   Scherer, Klaus ^b   Schuller, Björn ^b,c  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b UNIVERSITY OF GENEVA   (Switzerland)

^c IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Decision level fusion; Emotion recognition; Lip activity detection; Multimedia; Voice activity detection

Indexed keywords

INTERACTIVE COMPUTER SYSTEMS;

ACTIVITY DETECTION; DECISION LEVEL FUSION; EMOTION RECOGNITION; MULTIMEDIA; VOICE ACTIVITY DETECTION;

SPEECH RECOGNITION;

EID: 84947261353     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2663204.2666271     Document Type: Conference Paper

References (26)

1. A. Abel, A. Hussain, Q. D. Nguyen, F. Ringeval, M. Chetouani, and M. Milgram. Maximising Audiovisual Correlation with Automatic Lip Tracking and Vowel Based Segmentations. Biometric ID Management and Multimodal Communication, joint COST 2101 and 2102 International Conference, Madrid, Spain, September 16-18 2009, Lecture Notes in Computer Science, 5707:65{72, 2009.
2. T. Banziger, M. Mortillaro, and K. Scherer. Introducing the Geneva Multimodal Expression Corpus for Experimental Research on Emotion Perception. Emotion, 12(5):1161{1179, 2012.
3. D. Bone, C.-C. Lee, and S. S. Narayanan. Robust Unsupervised Arousal Rating: A Rule-Based Framework with Knowledge-Inspired Vocal Features. IEEE Transactions on Affective Computing (TAC), 5(2):201{213, April-June 2014.
4. F. Burkhardt, A. Paeschke, M. Rolfes, W. F. Sendlmeier, and B. Weiss. A Database of German Emotional Speech. In Proc. of INTERSPEECH 2005, pages 1517{1520, Lisbon, Portugal, 2005. ISCA.
5. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor. Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine, 18(1):32{80, January 2001.
6. A. Dhall, R. Goecke, J. Joshi, K. Sikka, and T. Gedeon. Emotion Recognition in the Wild Challenge 2014: Baseline, Data and Protocol. In Proc. of ICMI 2014, Istanbul, Turkey, November 2014. ACM.
7. A. Dhall, R. Goecke, J. Joshi, M. Wagner, and T. Gedeon. Emotion recognition in the wild challenge 2013. In Proc. of ICMI 2013, pages 509{516, Sydney, Australia, December 2013. ACM.
8. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon. Static Facial Expression Analysis in Tough Conditions: Data, Evaluation Protocol and Benchmark. In Proc. of ICCV Workshops 2011, pages 2106{2112, Barcelona, Spain, November 2011. IEEE.
9. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon. Collecting Large, Richly Annotated Facial-Expression Databases from Movies. MultiMedia, IEEE, 19(3):34{41, July-September 2012.
10. F. Eyben, F. Weninger, and B. Schuller. Affect Recognition in Real-Life Acoustic Conditions { A New Perspective on Feature Selection. In Proc. of INTERSPEECH 2013, pages 2044{2048, Lyon, France, August 2013. ISCA.
11. F. Eyben, F. Weninger, S. Squartini, and B. Schuller. Real-life Voice Activity Detection with LSTM Recurrent Neural Networks and an Application to Hollywood Movies. In Proc. of ICASSP 2013, pages 483{487, Vancouver, Canada, May 2013. IEEE.
12. F. Eyben, M. Wollmer, and B. Schuller. openSMILE The Munich Versatile and Fast Open-Source Audio Feature Extractor. In Proc. of ACM Multimedia 2010, pages 1459{1462, Florence, Italy, 2010. ACM.
13. M. Grimm, K. Kroschel, and S. Narayanan. The Vera am Mittag German audio-visual emotional speech database. In Proc. of ICME 2008, pages 865{868, Hannover, Germany, June 2008. IEEE.
14. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The WEKA Data Mining Software: An Update. ACM SIGKDD Explorations Newsletter, 11(1):10{18, June 2009.
15. H. Hermansky. Perceptual Linear Predictive (PLP) Analysis of Speech. Journal of the Acoustical Society of America (JASA), 87(4):1738{1752, November 1990.
16. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder. The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent. IEEE Transactions on Affective Computing (TAC), 3(1):5{17, January-March 2012.
17. F. Ringeval, A. S.onderegger, J. Sauer, and D. Lalanne. Introducing the RECOLA Multimodal Corpus of Remote Collaborative and A-ective Interactions. In Proc. of EmoSPACE 2013, held in conjunction with FG 2013, Shanghai, China, April 2013. IEEE.
18. B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. Muller, and S. Narayanan. The INTERSPEECH 2010 Paralinguistic Challenge. In Proc. of INTERSPEECH 2010, pages 2794{2797, Makuhari, Japan, 2010. ISCA.
19. B. Schuller, S. Steidl, A. Batliner, J. Epps, F. Eyben, F. Ringeval, E. Marchi, and Y. Zhang. The INTERSPEECH 2014 Computational Paralinguistics Challenge: Cognitive & Physical Load. In Proc. of INTERSPEECH 2014, Singapore, Singapore, September 2014. ISCA.
20. B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, and S. Kim. The INTERSPEECH 2013 Computational Paralinguistics Challenge: Social Signals, Conict, Emotion, Autism. In Proc. of INTERSPEECH 2013, pages 148{152, Lyon, France, August 2013. ISCA.
21. B. Schuller, M. Valstar, R. Cowie, and M. Pantic. AVEC 2012: The Continuous Audio/Visual Emotion Challenge { An Introduction. In Proc. of ICMI 2012, pages 361{362, Santa Monica, CA, USA, October 2012. ACM.
22. B. Schuller, B. Vlasenko, F. Eyben, G. Rigoll, and A. Wendemuth. Acoustic Emotion Recognition: A Benchmark Comparison of Performances. In Proc. of ASRU 2009, pages 552{557, Merano, Italy, December 2009. IEEE.
23. F. Weninger, F. Eyben, B. W. Schuller, M. Mortillaro, and K. R. Scherer. On the Acoustics of Emotion in Audio: What Speech, Music and Sound have in Common. Frontiers in Emotion Science, 4(Article ID 292):1{12, May 2013.
24. X. Xiong and F. De la Torre. Supervised Descent Method and its Applications to Face Alignment. In Proc. of CVPR 2013, pages 532{539, Portland, OR, USA, 2012. IEEE.
25. G. Zhao and M. Pietikainen. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 29(6):915{928, June 2007.
26. X. Zhu and D. Ramanan. Face Detection, Pose Estimation, and Landmark Localization in the Wild. In Proc. of CVPR 2012, pages 2879{2886, Providence, RI, USA, 2012. IEEE.