Analysis of emotional effect on speech-body gesture interplay

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Volumn , Issue , 2014, Pages 1934-1938

  Yang, Zhaojun ^a   Narayanan, Shrikanth ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Body gesture; Emotion attributes; Mutual information; Speech prosody; Speech gesture interplay

Indexed keywords

CHEMICAL ACTIVATION; CONTINUOUS SPEECH RECOGNITION; SPEECH;

ACOUSTIC FEATURES; BODY GESTURE; EMOTION ATTRIBUTES; GESTURE INTERACTION; MUTUAL INFORMATIONS; SPEECH PROSODY; UPPER BODIES; VOCAL-TRACTS;

SPEECH COMMUNICATION;

EID: 84910069942     PISSN: 2308457X     EISSN: 19909772     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. D. McNeill, Hand and mind: What gestures reveal about thought. University of Chicago Press, 1992.
2. F. Quek, D. McNeill, R. Bryll, S. Duncan, X.-F. Ma, C. Kirbas, K. E. McCullough, and R. Ansari, "Multimodal human discourse: gesture and speech, " ACM Transactions on Computer-Human Interaction (TOCHI), vol. 9, no. 3, pp. 171-193, 2002.
3. P. Bernardis and M. Gentilucci, "Speech and gesture share the same communication system, " Neuropsychologia, vol. 44, no. 2, pp. 178-190, 2006.
4. S. D. Kelly, C. Kravitz, and M. Hopkins, "Neural correlates of bimodal speech and gesture comprehension, " Brain and language, vol. 89, no. 1, pp. 253-260, 2004.
5. J. Cassell, C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Becket, B. Douville, S. Prevost, and M. Stone, "Animated conversation: Rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents, " in Proceedings of the 21st annual conference on Computer graphics and interactive techniques. ACM, 1994, pp. 413-420.
6. S. Levine, C. Theobalt, and V. Koltun, "Real-time prosody-driven synthesis of body language, " in ACM Transactions on Graphics, vol. 28, no. 5. ACM, 2009, p. 172.
7. S. Yildirim, M. Bulut, C. M. Lee, A. Kazemzadeh, Z. Deng, S. Lee, S. Narayanan, and C. Busso, "An acoustic study of emotions expressed in speech." in INTERSPEECH, 2004.
8. P. N. Juslin and K. R. Scherer, "Vocal expression of affect, " The new handbook of methods in nonverbal behavior research, pp. 65- 135, 2005.
9. C.-M. Lee, S. Narayanan, and R. Pieraccini, "Recognition of negative emotions from the speech signal, " in Proc. of ASRU, 2001, pp. 240-243.
10. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, "Emotion recognition in human-computer interaction, " Signal Processing Magazine, IEEE, vol. 18, no. 1, pp. 32-80, 2001.
11. G. Castellano, L. Kessous, and G. Caridakis, "Emotion recognition through multiple modalities: face, body gesture, speech, " in Affect and emotion in human-computer interaction. Springer, 2008, pp. 92-103.
12. A. Metallinou, A. Katsamanis, and S. Narayanan, "Tracking continuous emotional trends of participants during affective dyadic interactions using body language and speech information, " Image and Vision Computing, Special Issue on Continuous Affect Analysis, 2012.
13. Z. Yang, A. Metallinou, E. Erzin, and S. Narayanan, "Analysis of interaction attitudes using data-driven hand gesture phrases, " in Proc. of ICASSP, 2014.
14. C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motion in expressive speech animation: Analysis and synthesis, " Audio, Speech, and Language Processing, IEEE Transactions on, vol. 15, no. 3, pp. 1075-1086, 2007.
15. C. Busso and S. Narayanan, "Interrelation between speech and facial gestures in emotional utterances: A single subject study, " IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2331-2347, 2007.
16. A. Metallinou, C.-C. Lee, C. Busso, S. Carnicke, and S. Narayanan, "The USC CreativeIT database: A multimodal database of theatrical improvisation, " in Proc. of Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality (MMC), 2010.
17. S. M. Carnicke, Stanislavsky in Focus: An Acting Master for the Twenty-First Century. Routledge, UK, 2008.
18. I. Guide, "AutodeskR, " 2008.
19. M. Sargin, Y. Yemez, E. Erzin, and A. Tekalp, "Analysis of head gesture and prosody patterns for prosody-driven head-gesture animation, " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 8, pp. 1330-1345, 2008.
20. P. Boersma, "Praat, a system for doing phonetics by computer, " Glot international, vol. 5, no. 9/10, pp. 341-345, 2002.
21. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schröder, "'feeltrace': An instrument for recording perceived emotion in real time, " in ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion, 2000.
22. A. Metallinou and S. Narayanan, "Annotation and processing of continuous emotional attributes: Challenges and opportunities, " in Automatic Face and Gesture Recognition (FG), IEEE International Conference and Workshops on, 2013, pp. 1-8.
23. J. M. Iverson and E. Thelen, "Hand, mouth and brain. the dynamic emergence of speech and gesture, " Journal of Consciousness Studies, vol. 6, no. 11-12, pp. 11-12, 1999.
24. A. V. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, "A coupled hmm for audio-visual speech recognition, " in Acoustics, Speech, and Signal Processing (ICASSP), IEEE International Conference on, vol. 2, 2002, pp. II-2013.
25. H. C. Yehia, T. Kuratate, and E. Vatikiotis-Bateson, "Linking facial animation, head motion and speech acoustics, " Journal of Phonetics, vol. 30, no. 3, pp. 555-568, 2002.