Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks

AVEC 2015 - Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, co-Located with MM 2015

Volumn , Issue , 2015, Pages 73-80

  He, Lang ^a   Jiang, Dongmei ^a   Yang, Le ^a   Pei, Ercheng ^a   Wu, Peng ^b   Sahli, Hichem ^b,c  

^a NORTHWESTERN POLYTECHNICAL UNIVERSITY   (China)

^b VRIJE UNIVERSITEIT BRUSSEL   (Belgium)

^c IMEC   (Belgium)

Author keywords

Audio and video features; DBLSTM RNN; Multimodal fusion; Offset; Physiological feature

Indexed keywords

BRAIN; CORRELATION METHODS; ELECTROCARDIOGRAPHY; FORECASTING; PHYSIOLOGY; RECURRENT NEURAL NETWORKS; SIGNAL PROCESSING;

AUDIO AND VIDEO; DBLSTM-RNN; MULTI-MODAL FUSION; OFFSET; PHYSIOLOGICAL FEATURES;

FEATURE EXTRACTION;

EID: 84960882502     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2808196.2811641     Document Type: Conference Paper

References (40)

1. F. Agrafioti, D. Hatzinakos, and A. K. Anderson. ECG pattern analysis for emotion detection. IEEE Transactions on Affective Computing, 3(1):102-115, 2012.
2. L. Chao, J. Tao, M. Yang, Y. Li, and Z. Wen. Multi-scale temporal modelling for dimensional emotion recognition in video. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 11-18. ACM, 2014.
3. T. Costa, D. Galati, and E. Rognoni. The hurst exponent of cardiac response to positive and negative emotional film stimuli using wavelet. Autonomic Neuroscience, 151(2):183-185, 2009.
4. F. Eyben, F. Weninger, F. Gross, and B. Schuller. Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In Proc. of ACM MM, pages 835-838. ACM, 2013.
5. Y. Fan, Y. Qian, F. Xie, and F. K. Soong. TTS synthesis with bidirectional LSTM based recurrent neural networks. In Proc. Interspeech, 2014.
6. R. Fernandez, A. Rendel, B. Ramabhadran, and R. Hoory. Prosody contour prediction with long short-term memory, bi-directional, deep recurrent neural networks. In Proceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), 2014.
7. A. Graves, N. Jaitly, and A.-r. Mohamed. Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273-278. IEEE, 2013.
8. A. Graves, A.-r. Mohamed, and G. Hinton. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 6645-6649. IEEE, 2013.
9. X. Guo. Study of emotion recognition based on electrocardiogram and RBF neural network. Procedia Engineering, 15:2408-2412, 2011.
10. R. Gupta, N. Malandrakis, B. Xiao, T. Guha, M. Van Segbroeck, M. Black, A. Potamianos, and S. Narayanan. Multimodal prediction of affective dimensions and depression in human-computer interactions. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 33-40. ACM, 2014.
11. H.Gunes, M. A. Nicolaou, and M. Pantic. Continuous analysis of affect from voice and face. Computer Analysis of Human Behaviour. Springer Verlag, London, pages 255-291, 2011.
12. A. Jan, H. Meng, Y. Gaus, F. Zhang, and S. Turabzadeh. Hilbert-Huang transform based physiological signals analysis for emotion recognition. In Signal Processing and Information Technology (ISSPIT), 2009 IEEE International Symposium on. IEEE, 2009.
13. A. Jan, H. Meng, Y. Gaus, F. Zhang, and S. Turabzadeh. Automatic depression scale prediction using facial expression dynamics and regression. In Proceedings of the 4th ACM international workshop on Audio/visual emotion challenge, pages 73-80. ACM, 2014.
14. J. Kim and E. Andre. Emotion recognition based on physiological changes in music listening. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(12):2067-2083, 2008.
15. S. D. Kreibig. Autonomic nervous system activity in emotion: A review. Biological Psychology, 84(3):394-421, 2010.
16. B. Mathieu, S. Essid, T. Fillon, J. Prado, and G. Richard. YAAFE, an easy to use and efficient audio feature extraction software. Proceedings of Ismir Conference, 2010.
17. H. Meng, D. Huang, H. Wang, H. Yang, M. Al-Shuraifi, and Y. Wang. Depression recognition based on dynamic facial and vocal expression features using partial least square regression. In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pages 21-29. ACM, 2013.
18. V. Mitra, E. Shriberg, M. McLaren, A. Kathol, C. Richey, D. Vergyri, and M. Graciarena. The SRI AVEC-2014 evaluation system. In Proceedings of the 4th ACM international workshop on Audio/visual emotion challenge, pages 93-101. ACM, 2014.
19. M. A. Nicolaou, H. Gunes, and M. Pantic. Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. Affective Computing, IEEE Transactions on, 2(2):92-105, 2011.
20. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani. Robust continuous prediction of human emotions using multiscale dynamic cues. In Proceedings of the 14th ACM international conference on multimodal interaction, pages 501-508. ACM, 2012.
21. P. Prenter. Splines and Variational Methods. Wiley, New York, 1989.
22. P. Pudil, J. Novovičova, and J. Kittler. Floating search methods in feature selection. Pattern recognition letters, 15(11):1119-1125, 1994.
23. F. Ringeval, B. Schuller, M. Valstar, S. Jaiswal, E. Marchi, D. Lalanne, R. Cowie, and M. Pantic. AV+EC 2015 - The first affect recognition challenge bridging across audio, video, and physiological data. In Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge (AVEC), ACM MM, Brisbane, Australia, October 2015.
24. F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne. Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions. In Proceedings of Face and Gestures 2013, 2nd IEEE International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE), Shanghai, China, April 2013.
25. H. Sak, A. Senior, and F. Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Proceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), 2014.
26. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic. AVEC 2011-the first international audio/visual emotion challenge. In Affective Computing and Intelligent Interaction, pages 415-424. Springer, 2011.
27. B. Schuller, M. Valster, F. Eyben, R. Cowie, and M. Pantic. AVEC 2012: The continuous audio/visual emotion challenge. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 449-456. ACM, 2012.
28. G. Valenza, P. Allegrini, A. Lanata, and E. P. Scilingo. Dominant Lyapunov exponent and approximate entropy in heart rate variability during emotional visual elicitation. Frontiers in Neuroengineering, 5(3), 2012.
29. G. Valenza, L. Citi, A. Lanata, E. Scilingo, and R. Barbieri. Revealing real-time emotional responses: A personalized assessment based on heartbeat dynamics. Nature-SCIENTIFIC REPORTS, 2014.
30. G. Valenza, A. Lanata, and E. Scilingo. The role of nonlinear dynamics in affective valence and arousal recognition. Affective Computing, IEEE Transactions On, 3(2):237-249, 2012.
31. M. Valstar, B. Schuller, K. Smith, T. Almaev, F. Eyben, J. Krajewski, R. Cowie, and M. Pantic. AVEC 2014: 3d dimensional affect and depression recognition challenge. In Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, pages 3-10. ACM, 2014.
32. M. Valstar, B. Schuller, K. Smith, F. Eyben, B. Jiang, S. Bilakhia, S. Schnieder, R. Cowie, and M. Pantic. AVEC 2013: The continuous audio/visual emotion and depression recognition challenge. In Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pages 3-10. ACM, 2013.
33. L. Van Der Maaten. Audio-visual emotion challenge 2012: A simple approach. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 473-476. ACM, 2012.
34. D. Ververidis and C. Kotropoulos. Fast sequential floating forward selection applied to emotional speech features estimated on des and susas data collections. In Proc. XIV European Signal Processing Conf, 2006.
35. M. Wollmer, M. Kaiser, F. Eyben, B. Schuller, and G. Rigoll. LSTM-modeling of continuous emotions in an audiovisual affect recognition framework. Image and Vision Computing, 31(2):153-163, 2013.
36. N. Wang, E. Ambikairajah, B. Celler, and N. Lovell. Accelerometry based classification of gait patterns using empirical mode decomposition. In Proc. of ICASSP, pages 617-620. IEEE, 2008.
37. F. Weninger, J. Bergmann, and B. Schuller. Introducing CURRENNT-the Munich open-source CUDA recurrent neural network toolkit. Journal of Machine Learning Research, 15, 2014.
38. F. Weninger, J. Geiger, M. Wollmer, B. Schuller, and G. Rigoll. Feature enhancement by deep LSTM networks for ASR in reverberant multisource environments. Computer Speech and Language, 28(4):888-902, 2014.
39. M. Wollmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie. Abandoning emotion classes-Towards continuous emotion recognition with modelling of long-range dependencies. In INTERSPEECH, pages 597-600, 2008.
40. M. Wollmer, B. Schuller, F. Eyben, and G. Rigoll. Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening. Selected Topics in Signal Processing, IEEE Journal of, 4(5):867-881, 2010.