Human perception of audio-visual synthetic character emotion expression in the presence of ambiguous and conflicting information

IEEE Transactions on Multimedia

Volumn 11, Issue 5, 2009, Pages 843-855

  Mower, Emily ^a   Mataric, Maja J ^a   Narayanan, Shrikanth ^a  

^a University of Southern California University Park ^*  (United States)

Author keywords

Expressive animation; Multimodality; Synthesis

Indexed keywords

AUDIO AND VIDEO; AUDIO INFORMATION; AUDIO-VISUAL; AUDIO-VISUAL FEATURES; BASIC EMOTIONS; EMOTION EXPRESSION; EXPRESSIVE ANIMATION; FEATURE SETS; HUMAN PERCEPTION; HUMAN VOICE; HUMANOID ROBOT; MULTIMODALITY; SYNTHESIS; SYNTHETIC CHARACTER; TWO CHANNEL; VIDEO DATA;

ANTHROPOMORPHIC ROBOTS; VIDEO RECORDING;

ANIMATION;

EID: 68549090714     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2009.2021722     Document Type: Article

References (41)

1. J. Gratch and S.Marsella, "A domain-independent framework for modeling emotion," Cogn. Syst. Res., vol. 5, no. 4, pp. 269-306, 2004.
2. R. W. Picard, Affective Computing. Cambridge, MA: MIT Press, 1997.
3. W. Swartout, J. Gratch, R. Hill, E. Hovy, S. Marsella, J. Rickel, and D. Traum, "Toward virtual humans," AI Mag., vol. 27, no. 2, pp. 96-108, 2006.
4. C. Breazeal, "Emotion and sociable humanoid robots," Int. J. Human-Comput. Stud., vol. 59, no. 1-2, pp. 119-155, 2003.
5. The Official Mr. Bill Homepage, Dreamsite Productions. [Online]. Available: http://www.mrbill.com/.
6. B. DeGelder and P. Bertelson, "Multisensory integration, perception, and ecological validity," Trends Cogn. Sci., vol. 7, no. 10, pp. 460-467, Oct. 2003.
7. B. de Gelder, "The perception of emotions by ear and by eye," Cogn. Emotion, vol. 14, no. 3, pp. 289-311, 2000.
8. D. Massaro, "Fuzzy logical model of bimodal emotion perception: Comment on" The perception of emotions by ear and by eye" by de Gelder and Vroomen," Cogn. Emotion, vol. 14, no. 3, pp. 313-320, 2000.
9. B. de Gelder, K. Böcker, J. Tuomainen, M. Hensen, and J. Vroomen, "The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses," Neurosci. Lett.w, vol. 260, no. 2, pp. 133-136, 1999.
10. J. Hietanen, J. Leppänen, M. Illi, and V. Surakka, "Evidence for the integration of audiovisual emotional information at the perceptual level of processing," Eur. J. Cogn. Psychol., vol. 16, no. 6, pp. 769-790, 2004.
11. S. Fagel, "Emotional McGurk effect," in Proc. Int. Conf. Speech Prosody, Dresden, Germany, 2006, vol. 1.
12. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 32-80, Jan. 2001.
13. C. M. Lee and S. Narayanan, "Towards detecting emotions in spoken dialogs," IEEE Trans. Speech Audio Process., vol. 13, no. 2, pp. 293-302, Mar. 2005.
14. C. Busso, Z. Deng, S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, "Analysis of emotion recognition using facial expressions, speech and multimodal information," in Proc. 6th Int. Conf. Multimodal Interfaces, State Park, PA, Oct. 2004, pp. 205-211.
15. T. Vogt and E. Andre, "Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition," in IEEE Int. Conf. Multimedia & Expo (ICME), Los Alamitos, CA, 2005, pp. 474-477.
16. M. Grimm, K. Kroschel, E. Mower, and S. Narayanan, "Primitives-based evaluation and estimation of emotions in speech," Speech Commun., vol. 49, pp. 787-800, 2007.
17. E. Mower, S. Lee, M. J. Matarić, and S. Narayanan, "Human perception of synthetic character emotions in the presence of conflicting and congruent vocal and facial expressions," in IEEE Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Las Vegas, NV, Apr. 2008, pp. 2201-2204.
18. E. Mower, S. Lee, M. J. Matarić, and S. Narayanan, "Joint-processing of audio-visual signals in human perception of conflicting synthetic character emotions," in Proc. IEEE Int. Conf. Multimedia & Expo (ICME), Hannover, Germany, 2008, pp. 961-964.
19. E. Mower, M. J. Matarić, and S. Narayanan, "Selection of emotionally salient audio-visual features for modeling human evaluations of synthetic character emotion displays," in Proc. Int. Symp. Multimedia (ISM), Berkeley, CA, Dec. 2008.
20. H. McGurk and J. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, 1976.
21. D. Mobbs, N. Weiskopf, H. C. Lau, E. Featherstone, R. J. Dolan, and C. D. Frith, "The Kuleshov effect: The influence of contextual framing on emotional attributions effect: The influence of contextual framing on emotional attributions," Social Cogn. Affect. Neurosci., vol. 1, no. 2, pp. 95-106, 2006.
22. H. K. M. Meeren, C. C. R. J. van Heijnsbergen, and B. de Gelder, "Rapid perceptual integration of facial expression and emotional body language," Proc. Nat. Acad. Sci., vol. 102, no. 45, pp. 16 518-16 523, 2005.
23. M. M. Bradley and P. J. Lang, "Measuring emotion: The self-assessment manikin and the semantic differential," J. Beh. Therapy Experiment. Psychiatry, vol. 25, no. 1, pp. 49-59, 1994.
24. F. Gosselin and P. G. Schyns, "Bubbles: A technique to reveal the use of information in recognition tasks," Vis. Res., vol. 41, no. 17, pp. 2261-2271, 2001.
25. M. Bulut, S. Lee, and S. Narayanan, "Recognition for synthesis: Automatic parameter selection for resynthesis of emotional speech from neutral speech," in Proc. Int. Conf. Acoustics, Speech, and Signal Processing (ICASSP), Las Vegas, NV, Apr. 2008, pp. 4629-4632.
26. S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan, "An acoustic study of emotions expressed in speech," in Proc. Int. Conf. Spoken Language Processing (ICSLP), Jeju Island, South Korea, 2004, pp. 2193-2196.
27. S. Sutton et al., "Universal speech tools: The cslu toolkit," in Proc. Int. Conf. Spoken Language Processing (ICSLP), Sydney, Australia, Nov.-Dec. 1998, pp. 3221-3224.
28. A. Hanjalic, "Extracting moods from pictures and sounds: Towards truly personalized TV," IEEE Signal Process. Mag., vol. 23, no. 2, pp. 90-100, Mar. 2006.
29. M. Grimm and K. Kroschel, "Rule-based emotion classification using acoustic features," in Proc. Conf. Telemedicine and Multimedia Communication, Kajetany, Poland, Oct. 2005.
30. M. Nicolao, C. Drioli, and P. Cosi, "Voice GMM modelling for FESTIVAL/MBROLA emotive TTS synthesis," in Proc. Int. Conf. Spoken Language Processing, Pittsburgh, PA, Sep. 17-21, 2006, pp. 1794-1797.
31. P. Ekman and W. Friesen, Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press, 1978.
32. P. Ekman, W. Friesen, and J. Hager, FACS. eBook. Salt Lake City, UT: Research Nexus, 2002.
33. V. Vapnik, Statistical Learning Theory. New York: Wiley, 1998.
34. I. Witten, E. Frank, L. Trigg, M. Hall, G. Holmes, and S. Cunningham, "Weka: Practical machine learning tools and techniques with Java implementations," in Proc. ANNES Int. Workshop Emerging Engineering and Connectionnist-Based Information Systems, Dunedin, New Zealand, 1999, vol. 99, pp. 192-196.
35. P. Oudeyer, "The production and recognition of emotions in speech: Features and algorithms," Int. J. Human-Comput. Stud., vol. 59, no. 1-2, pp. 157-183, 2003.
36. T. M. Mitchell, Machine Learning. New York: McGraw-Hill, 1997.
37. M. S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Recognizing facial expression: Machine learning and application to spontaneous behavior," in Proc. IEEE Comput. Soc. Conf. Computer Vision and Pattern Recognition, 2005, vol. 2, pp. 568-573.
38. Y. Lin and G. Wei, "Speech emotion recognition based on HMM and SVM," in Proc. Int. Conf. Machine Learning and Cybernetics, Aug. 2005, vol. 8, pp. 4898-4901.
39. P. Rani,C. Liu, and N. Sarkar, "An empirical study of machine learning techniques for affect recognition in human-robot interaction," Pattern Anal. Appl., vol. 9, no. 1, pp. 58-69, May 2006.
40. C. Busso and S. Narayanan, "Joint analysis of the emotional fingerprint in the face and speech: A single subject study," in Proc. IEEE Int. Workshop Multimedia Signal Processing (MMSP), Chania, Greece, Oct. 2007.
41. C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan, "IEMOCAP: Interactive emotional dyadic motion capture database," J. Lang. Resources Eval., to be published.