Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data

Pattern Recognition Letters

Volumn 66, Issue , 2015, Pages 22-30

  Ringeval, Fabien ^a,b   Eyben, Florian ^a   Kroupi, Eleni ^b   Yuce, Anil ^b   Thiran, Jean Philippe ^b   Ebrahimi, Touradj ^b   Lalanne, Denis ^c   Schuller, Björn ^a,d,e  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

^b EPFL   (Switzerland)

^c UNIVERSITY OF FRIBOURG   (Switzerland)

^d IMPERIAL COLLEGE LONDON   (United Kingdom)

^e UNIVERSITY OF PASSAU   (Germany)

Author keywords

Audiovisual and physiological data; Context learning long short term memory recurrent neural networks; Continuous affect analysis; Multi task learning; Multimodal fusion; Multitime resolution features extraction

Indexed keywords

ARTIFICIAL INTELLIGENCE; BRAIN; CONTINUOUS TIME SYSTEMS; FORECASTING; LEARNING ALGORITHMS; PHYSIOLOGY; SUPERVISED LEARNING; TIME DOMAIN ANALYSIS;

AFFECT ANALYSIS; FEATURES EXTRACTION; LONG SHORT-TERM MEMORY; MULTI-MODAL FUSION; MULTITASK LEARNING; PHYSIOLOGICAL DATA;

RECURRENT NEURAL NETWORKS;

EID: 84943197961     PISSN: 01678655     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.patrec.2014.11.007     Document Type: Article

References (51)

1. H. Gunes, and B. Schuller Categorical and dimensional affect analysis in continuous input: current trends and future directions Image Vis. Comput. 31 2 2013 120 136
2. A. Tversky Intransitivity of preferences Psychol. Rev. 76 1969 31 48
3. M. Nicolaou, H. Gunes, and M. Pantic Automatic segmentation of spontaneous data using dimensional labels from multiple coders Proceedings of the LREC 2010 Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing, Istambul, Turkey 2010 43 48
4. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani Robust continuous prediction of human emotions using multiscale dynamic cues Proceedings of the 14th ACM Interernational Conference on Multimodal Interaction (ICMI), Istambul, Turkey 2012 501 508
5. S. Mariooryad, and C. Busso Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations Proceedings of the 5th International Conference on Affective Computing and Intelligent Interactions (ACII), Geneva, Switzerland 2013 85 90
6. S. Mariooryad, and C. Busso Correcting time-continuous emotional labels by modeling the reaction lag of evaluators IEEE Trans. Affect. Comput. PP 2014 12
7. D. Ververidis, and C. Kotropoulos Emotional speech recognition: resources, features, and methods Speech Commun. 48 9 2006 1162 1181
8. B. Vlasenko, B. Schuller, A. Wendemuth, and G. Rigoll Frame vs. turn-level: emotion recognition from speech considering static and dynamic processing Proceedings of the 2nd ACM International Conference on Affective Computing and Intelligent Interaction (ACII), Lisbon, Portugal 2007 139 147
9. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. McRorie, J.C. Martin, L. Devillers, S. Abrilian, A. Batliner, N. Amir, and K. Karpouzis The HUMAINE database: addressing the collection and annotation of naturalistic and induced emotional data A. Paiva, R. Prada, R. Picard, 2nd International Conference on Affective Computing and Intelligent Interaction (ACII), LNCS 4738 2007 Springer-Verlag Berlin Heidelberg, Lisbon, Portugal 488 500
10. M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. ter Maat, G. McKeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M. Valstar, and M. Wöllmer Building autonomous sensitive artificial listeners IEEE Trans. Affect. Comput. 3 2 2012 165 183
11. M. Grimm, K. Kroschel, and S. Narayanan Support vector regression for automatic recognition of spontaneous emotions in speech Proc. of the 32th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Honolulu, HI, USA 4 2007 1085 1088
12. M. Chetouani, A. Mahdhaoui, and F. Ringeval Time-scale feature extractions for emotional speech characterisation Cogn. Comput. 1 2 2009 194 201
13. F. Eyben, M. Wöllmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie On-line emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues J. Multimodal User Interfaces 3 1-2 2010 7 12
14. H. Gunes, and M. Pantic Automatic, dimensional and continuous emotion recognition Int. J. Synth. Emotions 1 1 2010 68 90
15. R. Levenson Emotion and the autonomic nervous system: a prospectus for research on autonomic specificity H.L. Wagner, Social Psychophysiology and Emotion: Theory and Clinical Applications 1988 John Wiley & Sons 17 42
16. J. Kim Bimodal emotion recognition using speech and physiological changes M. Grimm, K. Kroschel, Robust Speech Recognition and Understanding 2007 I-Tech Education and Publishing Vienna, Austria 265 280
17. M. Nicolaou, H. Gunes, and M. Pantic Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space IEEE Trans. Affect. Comput. 2 2 2011 92 105
18. A. Metallinou, A. Katsamanis, Y. Wang, and S. Narayanan Tracking changes in continuous emotion states using body language and prosodic cues Proceedings of the 36th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2011 2288 2291
19. I. Kanluan, M. Grimm, and K. Kroschel Audio-visual emotion recognition using an emotion recognition space concept Proceedings of the 16th European Signal Processing Conference (EUSIPCO), Lausanne, Switzerland 2008
20. M. Nicolaou, H. Gunes, and M. Pantic Audio-visual classification and fusion of spontaneous affective data in likelihood space Proceedings of the 20th IEEE International Conference on Pattern Recognition (ICPR), Istambul, Turkey 2010 3695 3699
21. F. Eyben, M. Wöllmer, and B. Schuller A multi-task approach to continuous five-dimensional affect sensing in natural speech ACM Trans. Interact. Intell. Syst. (TiiS) - Spec. Issue Affect. Interact. Natural Environ. 2 1 2012
22. F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions Proceedings of Face & Gestures 2013, 2nd IEEE International Workshop on Emotion Representation, Analysis and Synthesis in Continuous Time and Space (EmoSPACE), Shanghai, China 2013
23. J. Hall, and W. Watson The effects of a normative intervention on group decision-making performance Hum. Relat. 23 4 1970 299 317
24. M. Grimm, K. Kroschel, and S. Narayanan The Vera am Mittag German audio-visual emotional speech database Proceedings of the 8th International Conference on Multimedia & Expo (ICME), Hannover, Germany 2008 865 868
25. L. Li A concordance correlation coefficient to evaluate reproducibility Biometrics 45 1 1989 255 268
26. B. Schuller, S. Steidl, A. Batliner, A. Vinciarelli, K. Scherer, F. Ringeval, M. Chetouani, F. Weninger, F. Eyben, E. Marchi, M. Mortillaro, H. Salamin, A. Polychroniou, F. Valente, and S. Kim The INTERSPEECH 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism Proceedings of INTERSPEECH 2013, 14th Annual Conference of the International Speech Communication Association (ISCA), Lyon, France 2013 148 152
27. B. Schuller, S. Steidl, A. Batliner, J. Epps, F. Eyben, F. Ringeval, E. Marchi, and Y. Zhang The INTERSPEECH 2014 computational paralinguistics challenge: cognitive & physical load Proceedings of INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association (ISCA), Singapore, Republic of Singapore 2014
28. F. Eyben, M. Wöllmer, and B. Schuller openSMILE - the Munich versatile and fast open-source audio feature extractor Proceedings of the 18th ACM International Conference on Multimedia (ACM MM), Florence, Italy 2010 1459 1462
29. F. Eyben, F. Weninger, F. Groß, and B. Schuller Recent developments in openSMILE, the Munich open-source multimedia feature extractor Proceedings of the 21st ACM International Conference on Multimedia (ACM MM), Barcelona, Spain 2013 835 838
30. F. Weninger, F. Eyben, B.W. Schuller, M. Mortillaro, and K.R. Scherer On the acoustics of emotion in audio: what speech, music and sound have in common Front. Psychol. Emotion Sci. Spec. Issue Express. Emot. Music Vocal Commun. 4 Article ID 292 2013 1 12
31. X. Xiong, and F. De la Torre Supervised descent method and its applications to face alignment Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Portland (OR), USA 2013 532 539
32. G. Farnebäck Two-frame motion estimation based on polynomial expansion Image Analysis, Special Issue, 13th Scandinavian Conference (SCIA) 2749 2003 Springer Halmstad, Sweden 363 370
33. S.C. Chen, C.H. Wu, S.Y. Lin, and Y.P. Hung 2D face alignment and pose estimation based on 3D facial models Proceedings of the 12th International Conference on Multimedia & Expo (ICME) 2012 IEEE Melbourne, Australia 128 133
34. P. Ekman, and W.V. Friesen Facial Action Coding System: A Technique for the Measurement of Facial Movement 1978 Consulting Psychologists Press Palo Alto, CA
35. D. Lowe Distinctive image features from scale-invariant keypoints Int. J. Comput. Vis. 60 2 2004 91 110
36. J. Xiao, T. Moriyama, T. Kanade, and J.F. Cohn Robust full-motion recovery of head by dynamic templates and re-registration techniques Int. J. Imaging Syst. Technol. 13 1 2003 85 94
37. R. Picard, E. Vyzas, and J. Healey Toward machine emotional intelligence: analysis of affective physiological state IEEE Trans. Pattern Anal. Mach. Intell. 23 10 2001 1175 1191
38. K.C. Bilchick, and R.D. Berger Heart rate variability J. Cardiovasc. Electrophysiol. 17 6 2006 691 694
39. J. Pan, and W.J. Tompkins A real-time QRS detection algorithm IEEE Trans. Biomed. Eng. 32 3 1985 230 236
40. J.M. Hausdorff, A. Lertratanakul, M.E. Cudkowicz, A. Peterson, D. Kaliton, and A.L. Golberger Dynamic markers of altered gait rhythm in amyotrophic lateral sclerosis J. Appl. Physiol. 88 2000 2045 2053
41. A. Kalauzi, T. Bojic, and L. Rakic Extracting complexity waveforms from one-dimensional signals Nonlinear Biomed. Phys. 3 2009
42. M. Dawson, A. Schell, and D. Filion The electrodermal system J.T. Cacioppo, L.G. Tassinary, G.G. Berntson, Handbook of psychophysiology 2 2007 Cambridge University Press Cambridge 200 223
43. F. Eyben Real-time Speech and Music Classification by Large Audio Feature Space Extraction 2014 Technische Universität München Ph.D. thesis
44. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber Gradient flow in recurrent nets: the difficulty of learning long-term dependencies S.C. Kremer, J.F. Kolen, Field Guide to Dynamical Recurrent Networks 2001 IEEE Press
45. S. Hochreiter, and J. Schmidhuber Long short-term memory Neural Comput. 9 8 1997 1735 1780
46. A. Graves, and J. Schmidhuber Framewise phoneme classification with bidirectional LSTM and other neural network architectures 18th International Joint Conference on Neural Networks (IJCNN), Neural Networks (special issue) 18 5-6 2005 602 610
47. M. Schuster, and K.K. Paliwal Bidirectional recurrent neural networks IEEE Trans. Signal Process. 45 11 1997 2673 2681
48. W.J. Yan, Q. Wu, J. Liang, Y.H. Chen, and W. Fu How fast are the leaked facial expressions: the duration of micro-expressions J. Nonverbal Behav. 37 4 2013 217 230
49. P. Werbos Backpropagation through time: what it does and how to do it Proc. IEEE 78 10 1990 1550 1560
50. A. Graves Supervised Sequence Labelling with Recurrent Neural Networks 2008 Technische Universität München, Munich, Germany Ph.D. thesis
51. D. Schneegaß, A.M. Schaefer, and T. Martinetz The intrinsic recurrent support vector machine Proceedings of the 15th IEEE European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium 2007 325 330