Continuous Estimation of Emotions in Speech by Dynamic Cooperative Speaker Models

IEEE Transactions on Affective Computing

Volumn 8, Issue 3, 2017, Pages 314-327

  Mencattini, Arianna ^a   Martinelli, Eugenio ^a   Ringeval, Fabien ^b   Schuller, Bjorn ^b,c   Natale, Corrado Di ^a  

^a UNIVERSITY OF ROME TOR VERGATA   (Italy)

^b UNIVERSITY OF PASSAU   (Germany)

^c IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

cooperative regression model; naturalistic emotional display; Speech emotion recognition

Indexed keywords

REGRESSION ANALYSIS;

AGGREGATION STRATEGY; AUTOMATIC EMOTION RECOGNITION; AUTOMATIC PREDICTION; MODULAR ARCHITECTURES; RECOGNITION SYSTEMS; REGRESSION MODEL; SPEECH EMOTION RECOGNITION; WEB-BASED APPLICATIONS;

SPEECH RECOGNITION;

EID: 85030083921     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2016.2531664     Document Type: Article

References (54)

1. S. Alghowinem, R. Goecke, M. Wagner, J. Epps, M. Breakspear, and G. Parker, "From joyous to clinically depressed: Mood detection using spontaneous speech, " in Proc. 25th Int. Florida Artif. Intell. Res. Soc. Conf., Marco Island, FL, USA, 2012, pp. 141-146.
2. K. E. B. Ooi, M. Lech, and N. B. Allen, "Prediction of major depression in adolescents using an optimized multi-channel weighted speech classification system, " Biomed. Signal Process. Control, vol. 14, pp. 228-239, Nov. 2014.
3. E. Marchi, F. Ringeval, and B. Schuller, "Voice-enabled assistive robots for handling autism spectrum conditions: An examination of the role of prosody, " in Speech and Automata in Health Care (Speech Technology and Text Mining in Medicine and Healthcare), A. Neustein, Ed., Boston, MA, USA: De Gruyter, 2014, pp. 207-236.
4. F. Ringeval, J. Demouy, G. Szaszak, M. Chetouani, L. Robel, J. Xavier, D. Cohen, and M. Plaza, "Automatic intonation recognition for prosodic assessment of language impaired children, " IEEE Trans. Audio, Speech Language Process., vol. 19, no. 5, pp. 1328-1342, Jul. 2011.
5. R. Looije, M. A. Neerincx, and F. Cnossen, "Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors, " Int. J. Human-Comput. Stud., vol. 68, no. 6, pp. 386-397, Jun. 2010.
6. A. Batliner, F. Burkhardt, M. van Ballegooy, and E. Noth, "A taxonomy of applications that utilize emotional awareness, " in Proc. 5th 1st Int. Conf. Slovenian Language Technol., Ljubljana, Slovenia, 2006, pp. 246-250.
7. Q. Luo, "Research on e-learning system using speech emotion recognition, " Adv. Sci. Lett., vol. 5, no. 1, pp. 363-366, Jan. 2012.
8. H. Lu, M. Rabbi, G. T. Chittaranjan, D. Frauendorfer, M. S. Mast, A. T. Campbell, D. Gatica-Perez, and T. Choudhury, "Stresssense: Detecting stress in unconstrained acoustic environments using smartphones, " in Proc. ACM 14th Int. Conf. Ubiquitous Comput., Pittsburgh, PA, USA, 2012, pp. 351-360.
9. C. Clavel, I. Vasilescu, L. Devillers, G. Richard, and T. Ehrette, "Fear-type emotion recognition for future audio-based surveillance systems, " Speech Commun., vol. 50, no. 6, pp. 487-503, Jun. 2008.
10. H. Gunes and B. Schuller, "Categorical and dimensional affect analysis in continuous input: Current trends and future directions, " Image Vis. Comput.: Affect Anal. Continuous Input, vol. 31, no. 2, pp. 120-136, Feb. 2013.
11. D. Ververidis and C. Kotropoulos, "Emotional speech recognition: Resources, features, and methods, " Speech Commun., vol. 48, no. 9, pp. 1162-1181, Sep. 2006.
12. M. E. Ayadi, M. Kamel, and F. Karray, "Survey on speech emotion recognition: Features, classification schemes, and databases, " Pattern Recog., vol. 44, no. 3, pp. 572-587, Mar. 2011.
13. A. Mencattini, E. Martinelli, G. Costantini, M. Todisco, B. Basile, M. Bozzali, and C. D. Natale, "Speech emotion recognition using amplitude modulation parameters and a combined feature selection procedure, " Knowl.-Based Syst., vol. 63, pp. 68-81, Jun. 2014.
14. A. Batliner, B. Schuller, D. Seppi, S. Steidl, L. Devillers, L. Vidrascu, T. Vogt, V. Aharonson, and N. Amir, "The automatic recognition of emotions in speech, " in Emotion-Oriented Systems: The HUMAINE Handbook, Cognitive Technologies, R. Cowie, P. Petta, and C. Pelachaud, Eds. New York, NY, USA: Springer, 2010, pp. 71-99.
15. R. Cowie and R. Cornelius, "Describing the emotional states that are expressed in speech, " Speech Commun., vol. 40, no. 1-2, pp. 5-32, Apr. 2003.
16. J. Posner, J. Russell, and B. Peterson, "The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology, " Develop. Psychopathol., vol. 17, no. 3, pp. 715-734, Sep. 2005.
17. E. Douglas-Cowie, "The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data, " in Proc. 2nd Int. Conf. Affective Comput. Intell. Interaction, Lisbon, Portugal, 2007, vol. 4738, pp. 488-500.
18. M. Schroder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. ter Maat, G. McKeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M. Valstar, and M. Wollmer, "Building autonomous sensitive artificial listeners, " IEEE Trans. Affective Comput., vol. 3, no. 2, pp. 165-183, Apr.-Jun. 2012.
19. F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne, "Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions, " in Proc. 2nd IEEE Int. Workshop Emotion Represent., Anal. Synthesis Continuous Time Space, Shanghai, China, Apr. 2013, p. 8.
20. M. Grimm, K. Kroschel, and S. Narayanan, "Support vector regression for automatic recognition of spontaneous emotions in speech, " in Proc. 32th IEEE Int. Conf. Acoust., Speech Signal Process, Honolulu, HI, USA, 2007, vol. 4, pp. 1085-1088.
21. M. Chetouani, A. Mahdhaoui, and F. Ringeval, "Time-scale feature extractions for emotional speech characterisation, " Cognitive Comput., vol. 1, no. 2, pp. 194-201, Jun. 2009.
22. F. Eyben, M. Wollmer, A. Graves, B. Schuller, E. Douglas-Cowie, and R. Cowie, "On-line emotion recognition in a 3-D activationvalence-time continuum using acoustic and linguistic cues, " J. Multimodal User Interfaces, vol. 3, no. 1/2, pp. 7-12, Mar. 2010.
23. K. R. Scherer, A. Schorr, and T. Johnstone, "Appraisal processes in emotion: Theory, methods, research, " in Series in Affective Science, K. R. Scherer, A. Schorr, and T. Johnstone, Eds. Oxford, London, UK: Oxford Univ. Press, 2001.
24. A. Tversky, "Intransitivity of preferences, " Psychol. Rev., vol. 76, pp. 31-48, Jan. 1969.
25. S. Mariooryad and C. Busso, "Correcting time-continuous emotional labels by modeling the reaction lag of evaluators, " IEEE Trans. Affective Comput., vol. 6, no. 2, pp. 97-108, Apr.-Jun. 2015.
26. F. Eyben, M. Wollmer, and B. Schuller, "A multi-task approach to continuous five-dimensional affect sensing in natural speech, " ACMTrans. Interactive Intell. Syst. (TiiS)-Special Issue Affective Interaction Natural Environ., vol. 2, no. 1, Art. no. 6, p. 29, Mar. 2012.
27. F. Ringeval, F. Eyben, E. Kroupi, A. Yuce, J.-P. Thiran, T. Ebrahimi, D. Lalanne, and B. Schuller, "Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data, " Pattern Recog. Lett., vol. 66, pp. 22-30, Nov. 2015.
28. M. Nicolaou, H. Gunes, and M. Pantic, "Automatic segmentation of spontaneous data using dimensional labels from multiple coders, " in Proc. Int. Workshop Multimodal Corpora: Adv. Capturing, Coding Anal., Istanbul, Turkey, 2010, pp. 43-48.
29. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multiscale dynamic cues, " in Proc. 14th ACM Int. Conf. Multimodal Interaction, Istanbul, Turkey, 2012, pp. 501-508.
30. S. Mariooryad and C. Busso, "Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations, " in Proc. 5th Humaine Assoc. Int. Conf. Affective Comput. Intell. Interactions, Geneva, Switzerland, Sep. 2013, pp. 85-90.
31. D. Wu, T. Parsons, E. Mower, and S. Narayanan, "Speech emotion estimation in 3D space, " in Proc. IEEE Int. Conf. Multimedia Expo, Suntec, Singapore, 2010, pp. 737-742.
32. M. Nicolaou, H. Gunes, and M. Pantic, "Output-associative RVM regression for dimensional and continuous emotion prediction, " Image Vis. Comput., vol. 30, no. 3, pp. 186-196, Mar. 2012.
33. F. Weninger, F. Eyben, B. Schuller, M. Mortillaro, and K. R., "On the acoustics of emotion in audio: What speech, music and sound have in common, " Frontiers Psychol., vol. 4, Art. no. 292, p. 12, May 2013.
34. Z. Zhang, F. Ringeval, B. Dong, E. Coutinho, E. Marchi, and B. Schuller, "Enhanced semi-supervised learning for multimodal emotion recognition, " in Proc. 41st IEEE Int. Conf. Acoustics, Speech Signal Process., Shanghai, China, Apr. 2016, to appear.
35. J. Mendes-Moreira, C. Soares, A. Jorge, and J. D. Sousa, "Ensemble approaches for regression: A survey, " ACM Comput. Surveys, vol. 45, no. 1, Art. no. 10, Nov. 2012.
36. Y. Fan, M. Xu, Z. Wu, and L. Cai, "Automatic emotion variation detection using multi-scaled sliding window, " in Proc. IEEE Int. Conf. Orange Technol., Xian, China, 2014, pp. 232-236.
37. G. K. Verma and U. S. Tiwary, "Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals, " NeuroImage, Special Issue Multimodal Data Fusion, vol. 102, no. 1, pp. 162-172, Nov. 2013.
38. F. Ringeval, et al., "AV+EC 2015-The first affect recognition challenge bridging across audio, video, and physiological data, " in Proc. 5th Int. Workshop Audio/Visual Emotion Challenge., Brisbane, Australia, Oct. 2015, pp. 3-8.
39. S. K. D'Mello and J. Kory, "A review and meta-analysis of multimodal affect detection systems, " ACM Comput. Surveys, vol. 47, no. 3, Apr. 2015.
40. E. Martinelli, G. Magna, A. Vergara, and C. D. Natale, "Cooperative classifiers for reconfigurable sensor arrays, " Sensors Actuators B, Chem., vol. 199, pp. 83-92, Aug. 2014.
41. M. Grimm, K. Kroschel, and S. Narayanan, "The Vera am mittag German audio-visual emotional speech database, " in Proc. 8th Int. Conf. Multimedia Expo., Hannover, Germany, 2008, pp. 865-868.
42. B. Schuller, et al., "The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism, " in Proc. 14th Annu. Conf. Int. Speech Commun. Assoc., Lyon, France, 2013, pp. 148-152.
43. F. Eyben, F. Weninger, F. Gross, and B. Schuller, "Recent developments in openSMILE, the Munich open-source multimedia feature extractor, " in Proc. ACM 21st Int. Conf. Multimedia, Barcelona, Spain, 2013, pp. 835-838.
44. F. Eyben, Real-Time Speech and Music Classification by Large Audio Feature Space Extraction. New York, NY, USA: Springer, 2016.
45. R. Banse and K. Scherer, "Acoustic profiles in vocal emotion, " J. Personality Social Pscychol., vol. 70, no. 3, pp. 614-636, Mar. 2009.
46. B. Schuller and A. Batliner, Computational Paralinguistics-Emotion, Affect, and Personality in Speech and Language Processing. New York, NY, USA: Wiley, Nov. 2013.
47. Y. Saeys, I. Inza, and P. Larranaga, "A review of feature selection techniques in bioinformatics, " Bioinformatics, vol. 23, no. 19, pp. 2507-2517, Aug. 2007.
48. H. Peng, F. Long, and C. Ding, "Feature selection based on mutual information criteria of max-dependency, max-relevance, and minredundancy, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, pp. 1226-1238, Aug. 2005.
49. S. de Jong, "Simpls: An alternative approach to partial least squares regression, " Chemometrics Intell. Laboratory Syst., vol. 18, no. 3, pp. 251-263, Mar. 1993.
50. I.-K. L. Lin, "A concordance correlation coefficient to evaluate reproducibility, " Biometrics, vol. 45, no. 1, pp. 255-268, Mar. 1989.
51. J. Jia, S. Zhang, F. Meng, Y. Wang, and L. Cai, "Emotional audiovisual speech synthesis based on PAD, " IEEE Trans. Audio, Speech Language Process., vol. 19, no. 3, pp. 570-582, Mar. 2011.
52. C.-C. Chang and C.-J. Lin, "LIBSVM: A library for support vector machines, " ACMTrans. Intell. Syst. Technol., vol. 2, no. 3, Art. no. 27, Apr. 2011.
53. F. Ringeval, E. Marchi, M. Mehu, K. Scherer, and B. Schuller, "Face reading from speech-predicting facial action units from audio cues, " in Proc. INTERSPEECH 2015 16th Annu. Conf. Int. Speech Commun. Assoc., Dresden, Germany, Sep. 2015, pp. 1977-1981.
54. A. Mencattini, F. Ringeval, B. Schuller, E. Martinelli, and C. D. Natale, "Continuous monitoring of emotions by a multimodal cooperative sensor system, " Procedia Eng., vol. 120, pp. 556-559, Jul. 2015.