Interrelation between speech and facial gestures in emotional utterances: A single subject study

IEEE Transactions on Audio, Speech and Language Processing

Volumn 15, Issue 8, 2007, Pages 2331-2347

  Busso, Carlos ^a   Narayanan, Shrikanth S ^a  

^a University of Southern California   (United States)

Author keywords

Affective state; Articulatory movements; Facial motion; Speech acoustic

Indexed keywords

ACOUSTIC FEATURES; ACOUSTIC SPEECH; AFFECTIVE STATE; ARTICULATORY MOVEMENTS; AUDIO-VISUAL DATABASE; COMMUNICATION CHANNELS; EMOTION RECOGNITION; FACE REGIONS; FACIAL ANIMATIONS; FACIAL FEATURES; FACIAL GESTURES; FACIAL MOTION; HUMAN COMMUNICATIONS; INTERNAL STRUCTURES; MAPPING PARAMETERS; MULTI MODALS; MULTI-LINEAR REGRESSIONS; NONVERBAL CHANNELS; PHONETIC PROPERTIES; PRINCIPAL COMPONENTS; SPECTRAL ENVELOPES; SPEECH FEATURES;

ACOUSTICS; ANIMATION; FACE RECOGNITION; LINGUISTICS; PHOTOMAPPING; PRINCIPAL COMPONENT ANALYSIS;

CORRELATION METHODS;

EID: 42949107237     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2007.905145     Document Type: Article

References (45)

1. D. McNeill, Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. Chicago Press, 1992.
2. E. Vatikiotis-Bateson, K. Munhall, Y. Kasahara, F. Garcia, and H. Yehia, "Characterizing audiovisual information during speech," in Proc. 4th Int. Conf. Spoken Lang. Process. (ICSLP 96), Philadelphia, PA, Oct. 1996, vol. 3, pp. 1485-1488.
3. J. Cassell, C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Bechet, B. Douville, S. Prevost, and M. Stone, "Animated conversation: Rule-based generation of facial expression gesture and spoken intonation for multiple conversational agents," in Comput. Graphics (Proc. ACM SIGGRAPH'94), Orlando, FL, 1994, pp. 413-420.
4. L. Valbonesi, R. Ansari, D. McNeill, F. Quek, S. Duncan, K. McCullough, and R. Bryll, "Multimodal signal analysis of prosody and hand motion: Temporal correlation of speech and gestures," in Proc. Eur. Signal Process. Conf. (EUSIPCO 02), Tolouse, France, Sep. 2002, pp. 75-78.
5. P. Ekman, "Facial expression and emotion," Amer. Psychol., vol. 48, no. 4, pp. 384-392, Apr. 1993.
6. P. Ekman and E. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). New York: Oxford Univ. Press, 1997.
7. C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motionin expressive speech animation: Analysis and synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 3, pp. 1075-1086, Mar. 2007.
8. R. Cowie and R. Cornelius, "Describing the emotional states that are expressed in speech," Speech Commun., vol. 40, no. 1-2, pp. 5-32, Apr. 2003.
9. K. Scherer, "Vocal communication of emotion: A review of research paradigms," Speech Commun., vol. 40, no. 1-2, pp. 227-256, Apr. 2003.
10. S. Yildirim, M. Bulut, C. Lee, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan, "An acoustic study of emotions expressed in speech," in Proc. 8th Int. Conf. Spoken Lang. Process. (ICSLP 04), Jeju Island, Korea, 2004, pp. 2193-2196.
11. P. Ekman, "About Brows: Emotional and Conversational Signals," in Human Ethology: Claims and Limits of a New Discipline, M. von Cranach, K. Foppa, W. Lepenies, and D. Ploog, Eds. New York: Cambridge Univ. Press, 1979, pp. 169-202.
12. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behavior," Speech Commun., vol. 26, no. 1-2, pp. 23-43, 1998.
13. H. Yehia, T. Kuratate, and E. Vatikiotis-Bateson, "Linking facial animation, head motion and speech acoustics," J. Phonetics, vol. 30, no. 3, pp. 555-568, Jul. 2002.
14. C. Cavé, I. Guaïtella, R. Bertrand, S. Santi, F. Harlay, and R. Espesser, "About the relationship between eyebrow movements and F0 variations," in Int. Conf. Spoken Lang. Process. (ICSLP), Philadelphia, PA, Oct. 1996, vol. 4, pp. 2175-2178.
15. C. Busso, Z. Deng, S. Yildirim, M. Bulut, C. Lee, A. Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, "Analysis of emotion recognition using facial expressions, speech and multimodal information," in Proc. 6th Int. Conf. Multimodal Interfaces ICMI 04., State College, PA, 2004, pp. 205-211.
16. J. Cassell, T. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjalmsson, and H. Yan, "Embodiment in conversational interfaces: Rea," in Proc. Int. Conf. Human Factors in Comput. Syst. (CHI 99), Pittsburgh, PA, May 1999, pp. 520-527.
17. W. Wahlster, A. Günter, R. Kruse, and B. Neumann, Eds., "Towards symmetric multi-modality: Fusion and fission of speech, gesture, and facial expression," in Proc. 26th German Conf. Artif. Intell., Berlin, Germany, 2003, pp. 1-18.
18. C. Busso, Z. Deng, U. Neumann, and S. Narayanan, "Natural head motion synthesis driven by acoustic prosodic features," Comput. Animation Virtual Worlds, vol. 16, no. 3-4, pp. 283-290, Jul. 2005.
19. H. P. Graf, E. Cosatto, V. Strom, and F. J. Huang, "Visual prosody: Facialmovements accompanying speech," in Proc. IEEE Int. Conf. Autom. Faces and Gesture Recognition, Washington, DC, May 2002, pp. 396-401.
20. E. Bevacqua and C. Pelachaud, "Expressive audio-visual speech," Comput. Animation Virtual Worlds, vol. 15, no. 3-4, pp. 297-304, Jul. 2004.
21. R. Gutierrez-Osuna, P. Kakumanu, A. Esposito, O. Garcia, A. Bojorquez, J. Castillo, and I. Rudomin, "Speech-driven facial animation with realistic dynamics," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 33-42, Feb. 2005.
22. S. Lee, S. Yildirim, A. Kazemzadeh, and S. Narayanan, "An articulatory studyof emotional speech production," in 9th Eur. Conf. Speech Commun. Technol. (Interspeech'05-Eurospeech), Lisbon, Portugal, Sep. 2005, pp. 497-500.
23. B. Granström and D. House, "Audiovisual representation of prosody inexpressive speech communication," Speech Commun., vol. 46, no. 3-4, pp. 473-484, Jul. 2005.
24. J. Jiang, A. Alwan, P. Keating, B. Chaney, E. Auer, Jr, and L. Bernstein, "On the relationship between face movements, tongue movements, and speech acoustics," EURASIP J. Appl. Signal Process., vol. 11, pp. 1174-1188, 2002.
25. J. P. Barker and F. Berthommier, "Estimation of speech acoustics from visual speech features: A comparison of linear and nonlinear models," in Conf. Audio-Visual Speech Process. (AVSP 99), Santa Cruz, CA, Aug. 1999, pp. 112-117.
26. Z. Zeng, J. Tu,M. Liu, T. Huang, B. Pianfetti, D. Roth, and S. Levinson, "Audio-visual affect recognition," IEEE Trans. Multimedia, vol. 9, no. 2, pp. 424-428, Feb. 2007.
27. G. Potamianos, C. Neti, G. Gravier,A. Garg, and A. Senior, "Recent advancesin the automatic recognition of audiovisual speech," Proc. IEEE, vol. 91, no. 9, pp. 1306-1326, Sep. 2003.
28. C. Conati and H. Mclaren, L. Ardissono, P. Brna, and A. Mitrovic, Eds., "Data-driven refinement of a probabilistic model of user affect," in Proc. 10th Int. Conf. User Modeling (UM2005), Berlin, Germany, 2005, pp. 40-49.
29. S. Kshirsagar and N. Magnenat-Thalmann, "A multilayer personality model," in Proc. 2nd Int. Symp. Smart Graphics (SMARTGRAPH 02), Hawthorne, NY, Jun. 2002, pp. 107-115.
30. Y. Li and H. Shum, "Learning dynamic audio-visual mapping with input-output midden Markov models," IEEE Trans. Multimedia, vol. 8, no. 3, pp. 542-549, Jun. 2006.
31. M. Nordstrand, G. Svanfeldt, B. Granström, and D. House, "Measurements of articulatory variations and communicative signals in expressive speech," in Audio Visual Speech Process. (AVSP 03), Sep. 2003, pp. 233-237.
32. E. M. Caldognetto, P. Cosi, C. Drioli, G. Tisato, and F. Cavicchio, "Coproduction of speech and emotions: Visual and acoustic modifications of some phonetic labial targets," in Audio Visual Speech Process. (AVSP 03), Sep. 2003, pp. 209-214.
33. J. Deller, J. Hansen, and J. Proakis, Discrete-Time Processing of Speech Signals. Piscataway, NJ: IEEE, 2000.
34. R. W. Picard, Affective Computing MIT Media Laboratory Perceptual Computing Section, Cambridge, MA, 1995, Tech. Rep. 321.
35. P. Boersma and D. Weeninck, Praat, a System for Doing Phonetics by Computer Inst. Phonetic Sci., Univ. Amsterdam, Amsterdam, The Netherlands, 1996, Tech. Rep. 132 [Online]. Available: http://www. praat.org
36. J. Jiang, A. Alwan, L. Bernstein, E. Auer, Jr, and P. Keating, "Predicting face movements from speech acoustics using spectral dynamics," in Proc. IEEE Int. Conf. Multimedia and Expo (ICME 02), Lausanne, Switzerland, Aug. 2002, vol. 1, pp. 181-184.
37. K. Arun, T. Huang, and S. Blostein, "Least-squares fitting of two 3-D pointsets," IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, no. 5, pp. 698-700, Sep. 1987.
38. E. Vatikiotis-Bateson and H. C. Yehia, "Speaking mode variability in multimodal speech production," IEEE Trans. Neural Netw., vol. 13, no. 4, pp. 894-899, Jul. 2002.
39. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book. Cambridge, U.K.: Entropic Cambridge Res. Lab., 2006.
40. T. Hazen, "Visual model structures and synchrony constraints for audio-visual speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 14, no. 3, pp. 1082-1089, May 2006.
41. C. Busso and S. Narayanan, "Interplay between linguistic and affective goalsin facial expression during emotional utterances," in Proc. 7th Int. Seminar Speech Prod. (ISSP 06), Ubatuba-SP, Brazil, Dec. 2006, pp. 549-556.
42. R. E. Kaliouby and P. Robinson, "Mind reading machines: Automated inference ofcognitive mental states from video," in IEEE Conf. Syst., Man, Cybern., The Hague, The Netherlands, Oct. 2004, vol. 1, pp. 682-688.
43. J. Cohn, L. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, "Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior," in Proc. IEEE Conf. Syst., Man, Cybern., The Hague, The Netherlands, Oct. 2004, vol. 1, pp. 610-616.
44. C. Lansing and G. McConkie, "Attention to facial regions in segmental and prosodic visual speech perception tasks," J. Speech, Lang., Hearing Res., vol. 42, pp. 526-539, Jun. 1999.
45. M. Swerts and E. Krahmer, "The importance of different facial areas for signalling visual prominence," in Proc. Int. Conf. Spoken Lang. Process. (ICSLP), Pittsburgh, PA, Sep. 2006, pp. 1280-1283.