Correcting time-continuous emotional labels by modeling the reaction lag of evaluators

IEEE Transactions on Affective Computing

Volumn 6, Issue 2, 2015, Pages 97-108

  Mariooryad, Soroosh ^a   Busso, Carlos ^a  

^a The University of Texas at Dallas   (United States)

Author keywords

emotion recognition; emotional descriptors; maximum mutual information; Time continuous emotion annotation

Indexed keywords

COMPUTER PROGRAMMING;

ACCURACY IMPROVEMENT; DESCRIPTORS; EMOTION CLASSIFICATION; EMOTION RECOGNITION; EMOTIONAL DIMENSIONS; MAXIMUM MUTUAL INFORMATION; MUTUAL INFORMATIONS; TIME-CONTINUOUS EMOTION ANNOTATION;

STEREOPHONIC BROADCASTING;

EID: 84930944930     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAFFC.2014.2334294     Document Type: Article

References (51)

1. C. Busso, M. Bulut, and S. Narayanan, "Toward effective auto matic recognition systems of emotion in speech," in Social Emo tions in Nature and Artifact: Emotions in Human and Human Computer Interaction, J. Gratch and S. Marsella, Eds. New York, NY, USA: Oxford Univ. Press, Nov. 2013.
2. R. Cowie and R. Cornelius, "Describing the emotional states that are expressed in speech," Speech Commun., vol. 40, no. 1-2, pp. 5-32, Apr. 2003.
3. A. Metallinou and S. Narayanan, "Annotation and processing of continuous emotional attributes: Challenges and opportunities," in Proc. 2nd Int. Workshop Emotion Representation, Anal. Synthesis Continuous Time Space, Shanghai, China, Apr. 2013, pp. 1-8.
4. R. Cowie, G. McKeown, and E. Douglas-Cowie, "Tracing emotion: An overview," Int. J. Synthesis Emotions, vol. 3, no. 1, pp. 1-17, Jan.-Jun. 2012.
5. H. Gunes, B. Schuller, M. Pantic, and R. Cowie, "Emotion representation, analysis and synthesis in continuous space: A survey," in Proc. IEEE Int. Conf. Automatic Face Gesture Recog., Santa Barbara, CA, USA, Mar. 2011, pp. 827-834.
6. H. Schlosberg, "Three dimensions of emotion," Psychol. Rev., vol. 61, no. 2, pp. 81-88, Mar. 1954.
7. J. Russell and A. Mehrabian, "Evidence for a three-factor theory of emotions," J. Res. Personality, vol. 11, no. 3, pp. 273-294, Sep. 1977.
8. J. Fontaine, K. Scherer, E. Roesch, and P. Ellsworth, "The world of emotions is not two-dimensional," Psychol. Sci., vol. 18, no. 12, pp. 1050-1057, Dec. 2007.
9. D. Grandjean, D. Sander, and K. Scherer, "Conscious emotional experience emerges as a function of multilevel, appraisal-driven response synchronization," Consciousness Cogn., vol. 17, no. 2, pp. 484-495, Jun. 2008.
10. A. Batliner, S. Steidl, and E. Noth, "Releasing a thoroughly annotated and processed spontaneous emotional database: The FAU Aibo emotion corpus," in Proc. 2nd Int. Workshop Emotion: Corpora Res. Emotion Affect, Int. Conf. Language Resources Eval., Philadelphia, PA, USA, May 2008, pp. 28-31.
11. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schroder, "'FEELTRACE': An instrument for recording perceived emotion in real time," in Proc. ISCA Tutorial Res. Workshop Speech Emotion, Sep. 2000, pp. 19-24.
12. F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne, "Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions," in Proc. 2nd Int. Workshop Emotion Representation, Anal. Synthesis Continuous Time Space, Shanghai, China, Apr. 2013, pp. 1-8.
13. S. Mariooryad and C. Busso, "Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations," in Proc. Humaine Assoc. Conf. Affective Comput. Intell. Interaction, Geneva, Switzerland, Sep. 2013, pp. 85-90.
14. K. Scherer, "Vocal communication of emotion: A review of research paradigms," Speech Commun., vol. 40, no. 1/2, pp. 227-256, Apr. 2003.
15. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, "The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent," IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 5-17, Jan.-Mar. 2012.
16. Y. Kim, E. Schmidt, and L. Emelle, "MoodSwings: A collaborative game for music mood label collection," in Proc. Int. Symp. Music Inf. Retrieval, Marrakech, Morocco, Sep. 2008, pp. 28-31.
17. F. Nagel, R. Kopiez, O. Grewe, and E. Altenmuller, "EMuJoy: Software for continuous measurement of perceived emotions in music," Behav. Res. Meth., vol. 39, no. 2, pp. 283-290, May 2007.
18. P. Ekman and W. Friesen, "Constants across cultures in the face and emotion," J. Personality Soc. Psychol., vol. 17, no. 2, pp. 124-129, Mar. 1971.
19. P. Ekman, W. Friesen, M. Sullivan, A. Chan, A. Diacoyanni-Tarlat-zis, K. Heider, R. Krause, W. LeCompte, T. Pitcairn, P. Ricci-Bitti, K. Scherer, and M. Tomita, "Universals and cultural differences in the judgments of facial expressions of emotion," J. Personality Soc. Psychol., vol. 53, pp. 712-717, Oct. 1987.
20. E. Mower, A. Metallinou, C.-C. Lee, A. Kazemzadeh, C. Busso, S. Lee, and S. Narayanan, "Interpreting ambiguous emotional expressions," in Proc. Int. Conf. Affective Comput. Intell. Interaction, Amsterdam, The Netherlands, Sep. 2009, pp. 1-8.
21. L. Vidrascu and L. Devillers, "Real-life emotions in naturalistic data recorded in a medical call center," in Proc. 1st Int. Workshop Emotion: Corpora Res. Emotion Affect (Int. Conf. Lang. Resource Eval.), Genoa, Italy, May 2006, pp. 20-24.
22. H. Gunes and M. Pantic, "Automatic, dimensional and continuous emotion recognition," Int. J. Synthesis Emotions, vol. 1, no. 1, pp. 68-99, Jan.-Jun. 2010.
23. C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan, "IEMOCAP: Interactive emotional dyadic motion capture database," J. Language Resource Eval., vol. 42, no. 4, pp. 335-359, Dec. 2008.
24. M. Grimm, K. Kroschel, and S. Narayanan, "The Vera am Mittag German audio-visual emotional speech database," in Proc. IEEE Int. Conf. Multimedia Expo., Hannover, Germany, Jun. 2008, pp. 865-868.
25. A. Metallinou, C.-C. Lee, C. Busso, S. Carnicke, and S. Narayanan, "The USC CreativeIT database: A multimodal database of theatrical improvisation," presented at the Workshop Multimodal Corpora: Adv. Capturing, Coding Analyzing Multimodality, Valletta, Malta, May 2010.
26. L. Devillers, R. Cowie, J. C. Martin, E. Douglas-Cowie, S. Abrilian, and M. McRorie, "Real life emotions in French and English TV video clips: An integrated annotation protocol combining continuous and discrete approaches," in Proc. Int. Conf. Lang. Resource Eval., Genoa, Italy, May 2006, pp. 1105-1110.
27. N. Malandrakis, A. Potamianos, G. Evangelopoulos, and A. Zlatintsi, "A supervised approach to movie emotion tracking," in Proc. IEEE Int. Conf. Acoustics, Speech Signal Process., Prague, Czech Republic, May 2011, pp. 2376-2379.
28. J. Nicolle, V. Rapp, K. Bailly, L. Prevost, and M. Chetouani, "Robust continuous prediction of human emotions using multi-scale dynamic cues," in Proc. Int. Conf. Multimodal Interaction, Santa Monica, CA, USA, Oct. 2012, pp. 501-508.
29. J. Arias, C. Busso, and N. Yoma, "Shape-based modeling of the fundamental frequency contour for emotion detection in speech," Comput. Speech Language, vol. 28, no. 1, pp. 278-294, Jan. 2014.
30. B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M. Pantic, "AVEC 2011-The first international audio/visual emotion challenge," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, vol. 6975/2011, pp. 415-424.
31. S. Mariooryad and C. Busso, "Exploring cross-modality affective reactions for audiovisual emotion recognition," IEEE Trans. Affect. Comput., vol. 4, no. 2, pp. 183-196, Apr.-Jun. 2013.
32. M. Wollmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, "Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies," in Proc. 9th Annu. Conf. Int. Speech Commun. Assoc., Brisbane, Australia, Sep. 2008, pp. 597-600.
33. J. C. Kim, H. Rao, and M. Clements, "Investigating the use of formant based features for detection of affective dimensions in speech," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, vol. 6975/2011, pp. 369-377.
34. H. Meng and N. Bianchi-Berthouze, "Naturalistic affective expression classification by a multi-stage approach based on hidden Markov models," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, vol. 6975/2011, pp. 378-387.
35. L. Cen, Z. L. Yu, and M. Dong, "Speech emotion recognition system based on L1 regularized linear regression and decision fusion," in Proc. 4th Int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, vol. 6975/2011, pp. 332-340.
36. M. Nicolaou, H. Gunes, and M. Pantic, "Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space," IEEE Trans. Affective Comput., vol. 2, no. 2, pp. 92-105, Apr.-Jun. 2011.
37. M. Nicolaou, V. Pavlovic, and M. Pantic, "Dynamic probabilistic CCA for analysis of affective behaviour," in Proc. Eur. Conf. Com-put. Vis., Florence, Italy, Oct. 2012, pp. 98-111.
38. M. Nicolaou, V. Pavlovic, and M. Pantic, "Dynamic probabilistic CCA for analysis of affective behaviour and fusion of continuous annotations," IEEE Trans. Pattern Anal. Mach. Intell., vol 36, no. 7, pp. 1299-1311, Jul. 2014.
39. E. Douglas-Cowie, R. Cowie, C. Cox, N. Amir, and D. Heylen, "The sensitive artificial listener: An induction technique for generating emotionally coloured conversation," in Proc. 2nd Int. Workshop Emotion: Corpora Res. Emotion Affect, Int. Conf. Lang. Resource Eval., Marrakech, Morocco, May 2008, pp. 1-8.
40. M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Automatic recognition of facial actions in spontaneous expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, Sep. 2006.
41. P. Ekman and W. Friesen, Facial Action Coding System: A Technique for Measurement of Facial Movement. Palo Alto, CA, USA: Consulting Psychologists Press, 1978.
42. B. Schuller, S. Steidl, A. Batliner, F. Schiel, and J. Krajewski, "The INTERSPEECH 2011 speaker state challenge," in Proc. 12th Annu. Conf. Int. Speech Commun. Assoc., Florence, Italy, Aug. 2011, pp. 3201-3204.
43. F. Eyben, M. Wollmer, and B. Schuller, "OpenSMILE: The Munich versatile and fast open-source audio feature extractor," in Proc. ACM Int. Conf. Multimedia, Florence, Italy, Oct. 2010, pp. 1459-1462.
44. J. Arias, C. Busso, and N. Yoma, "Energy and F0 contour modeling with functional data analysis for emotional speech detection," in Proc. Interspeech, Lyon, France, Aug. 2013, pp. 2871-2875.
45. G. Ramirez, T. Baltrusaitis, and L. Morency, "Modeling latent discriminative dynamic of multi-dimensional affective signals," in Proc. 4th int. Conf. Affective Comput. Intell. Interaction, Oct. 2011, vol. 6975/2011, pp. 396-406.
46. J. Russell, "A circumplex model of affect," J. Personality Soc. Psy-chol., vol. 39, no. 6, pp. 1161-1178, Dec. 1980.
47. C.-C. Lee, C. Busso, S. Lee, and S. Narayanan, "Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 1983-1986.
48. M. Wollmer, F. Eyben, B. Schuller, E. Douglas-Cowie, and R. Cowie, "Data-driven clustering in emotional space for affect recognition using discriminatively trained LSTM networks," in Proc. Interspeech, Brighton, U.K., Sep. 2009, pp. 1595-1598.
49. A. Metallinou, M. Wollmer, A. Katsamanis, F. Eyben, B. Schuller, and S. Narayanan, "Context-sensitive learning for enhanced audiovisual emotion classification," IEEE Trans. Affect. Comput., vol. 3, no. 2, pp. 184-198, Apr.-Jun. 2012.
50. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. Witten, "The WEKA data mining software: An update," ACM SIGKDD Explorations Newslett., vol. 11, no. 1, pp. 10-18, Jun. 2009.
51. W. Mendenhall and T. Sincich, Statistics for Engineering and the Sciences. Upper Saddle River, NJ, USA: Prentice-Hall, 2006.