Generating human-like behaviors using joint, speech-driven models for conversational agents

IEEE Transactions on Audio, Speech and Language Processing

Volumn 20, Issue 8, 2012, Pages 2329-2340

  Mariooryad, Soroosh ^a   Busso, Carlos ^a  

^a UNIVERSITY OF TEXAS AT DALLAS   (United States)

Author keywords

Conversational agent (CA); dynamic Bayesian network (DBN); facial animation; visual prosody

Indexed keywords

AUDIO-VISUAL INFORMATION; CONVERSATIONAL AGENTS; DYNAMIC BAYESIAN NETWORKS; FACIAL ANIMATION; FACIAL EXPRESSIONS; FACIAL GESTURES; HEAD MOTION; HUMAN COMMUNICATIONS; JOINT MODELS; PERCEPTUAL EVALUATION; VISUAL ASPECTS; VISUAL PERCEPTION; VISUAL PROSODY;

ANIMATION; BAYESIAN NETWORKS; MOTION PICTURE EXPERTS GROUP STANDARDS; SPEECH COMMUNICATION;

BEHAVIORAL RESEARCH;

EID: 84865398720     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2012.2201476     Document Type: Article

References (47)

1. C. Busso and S. Narayanan, "Joint analysis of the emotional fingerprint the face and speech: A single subject study," in Int. Workshop Multimedia Signal Process. (MMSP 2007), Chania, Crete, Greece, Oct. 2007, pp. 43-47.
2. K. Scherer, "Vocal communication of emotion: A review of research paradigms," Speech Commun., vol. 40, no. 1-2, pp. 227-256, Apr. 2003.
3. H. McGurk and J. W. MacDonald, "Hearing lips and seeing voices," Nature, vol. 264, pp. 746-748, Dec. 1976.
4. J. Cassell, C. Pelachaud, N. Badler, M. Steedman, B. Achorn, T. Bechet, B. Douville, S. Prevost, and M. Stone, "Animated conversation: Rule-based generation of facial expression gesture and spoken intonation for multiple conversational agents," in Proc. Comput. Graphics (Proc. ACM SIGGRAPH'94), Orlando, FL, 1994, pp. 413-420.
5. D. McNeill, Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
6. L. Valbonesi, R. Ansari, D. McNeill, F. Quek, S. Duncan, K. McCullough, and R. Bryll, "Multimodal signal analysis of prosody and hand motion: Temporal correlation of speech and gestures," in Proc. Eur. Signal Process. Conf. (EUSIPCO 02), Tolouse, France, Sep. 2002, pp. 75-78.
7. J. Cassell, T. Bickmore, L. Campbell, H. Vilhjálmsson, and H. Yan, "More than just a pretty face: Conversational protocols and the affordances of embodiment," Knowl.-Based Syst., vol. 14, pp. 55-64, Mar. 2001. (Pubitemid 32264619)
8. E. Marsi and F. van Rooden, "Expressing uncertainty with a talking head," in Proc.Workshop Multimodal Output Generation (MOG 2007), Aberdeen, U.K., Jan. 2007, pp. 105-116.
9. I. Poggi and C. Pelachaud, "Performative facial expressions animated faces," in Embodied Conversational Agents, J. Cassell, J. Sullivan, S. Prevost, and E. Churchill, Eds. Cambridge, MA:MIT Press, 2000, p. 154188.
10. M. Foster, "Comparing rule-based and data-driven selection of facial displays," in Proc. Workshop Embodied Lang. Process., Assoc. for Comput. Linguist., Prague, Czech Republic, Jun. 2007, pp. 1-8.
11. C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motion expressive speech animation: Analysis and synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 3, pp. 1075-1086, Mar. 2007.
12. J. Cassell, Y. Nakano, T. Bickmore, C. Sidner, and C. Rich, "Non-verbal cues for discourse structure," in Proc. 39th Annu. Meeting Assoc. for Comput. Linguist. (ACL2001), Toulouse, France, Jul. 2001, pp. 114-123.
13. M. Stone, D. DeCarlo, I. Oh, C. Rodriguez, A. Stere, A. Lees, and C. Bregler, "Speaking with hands: Creating animated conversational characters from recordings of human performance," ACM Trans. Graphics (TOG), vol. 23, pp. 506-513, August 2004.
14. C. Busso, Z. Deng, U. Neumann, and S. Narayanan, "Natural head motion synthesis driven by acoustic prosodic features," Comput. Animation and Virtual Worlds, vol. 16, no. 3-4, pp. 283-290, Jul. 2005. (Pubitemid 41495224)
15. C. Busso, Z. Deng, U. Neumann, and S. Narayanan, "Learning expressive human-like head motion sequences from speech," in Data-Driven 3D Facial Animations, Z. Deng and U. Neumann, Eds. Surrey, U.K.: Springer-Verlag, 2007, pp. 113-131.
16. C. Busso and S. Narayanan, "Interrelation between speech and facial gestures emotional utterances: A single subject study," IEEE Trans. Audio, Speech, Lang. Process., vol. 15, no. 8, pp. 2331-2347, Nov. 2007.
17. E. Vatikiotis-Bateson, K. Munhall, Y. Kasahara, F. Garcia, and H. Yehia, "Characterizing audiovisual information during speech," in Proc. 4h Int. Conf. Spoken Lang. Process. (ICSLP 96), Philadelphia, PA, Oct. 1996, vol. 3, pp. 1485-1488.
18. H. Yehia, P. Rubin, and E. Vatikiotis-Bateson, "Quantitative association of vocal-tract and facial behavior," Speech Commun., vol. 26, no. 1-2, pp. 23-43, 1998. (Pubitemid 128381217)
19. B. Granström and D. House, "Audiovisual representation of prosody expressive speech communication," Speech Commun., vol. 46, no. 3-4, pp. 473-484, Jul. 2005. (Pubitemid 40952529)
20. C. Cavé, I. Guaïtella, R. Bertrand, S. Santi, F. Harlay, and R. Espesser, "About the relationship between eyebrow movements and F0 variations," in Proc. Int. Conf. Spoken Lang. (ICSLP), Philadelphia, PA, Oct. 1996, vol. 4, pp. 2175-2178.
21. M. L. Flecha-García, "Eyebrowraises dialogue and their relation to discourse structure, utterance function and pitch accents English," Speech Commun., vol. 52, pp. 542-554, Jun. 2010.
22. H. P. Graf, E. Cosatto, V. Strom, and F. J. Huang, "Visual prosody: Facial movements accompanying speech," in Proc. of IEEE Int. Conf. Autom. Faces and Gesture Recognition,Washington, D. C., May 2002, pp. 396-401.
23. K. G. Munhall, J. A. Jones, D. E. Callan, T. Kuratate, and E. Vatikiotis-Bateson, "Visual prosody and speech intelligibility: Head movement improves auditory speech perception," Psychol. Sci., vol. 15, no. 2, pp. 133-137, February 2004. (Pubitemid 38361306)
24. J. Cassell, T. Bickmore, M. Billinghurst, L. Campbell, K. Chang, H. Vilhjalmsson, and H. Yan, "Embodiment conversational interfaces: Rea," in Proc. Int. Conf. Human Factors Comput. Syst. (CHI-99), Pittsburgh, PA, May 1999, pp. 520-527.
25. C. Pelachaud, N. Badler, and M. Steedman, "Generating facial expressions for speech," Cognitive Sci., vol. 20, no. 1, pp. 1-46, January 1996. (Pubitemid 126159862)
26. E. Bevacqua, M. Mancini, R. Niewiadomski, and C. Pelachaud, "An expressive ECA showing complex emotions," in Proc. Artif. Intell. and Simulation of Behaviour (AISB 2007) Annu. Conv., Newcastle, U.K., Apr. 2007, pp. 208-216.
27. D. DeCarlo, C. Revilla, M. Stone, and J. Venditti, "Making discourse visible: Coding and animating conversational facial displays," in Proc. Comput. Animat. (CA 2002), Geneva, Switzerland, Jun. 2002, pp. 11-16.
28. H. Yehia, T. Kuratate, and E. Vatikiotis-Bateson, "Facial animation and head motion driven by speech acoustics," in Proc. 5th Seminar Speech Production: Models and Data, Kloster Seeon, Bavaria, Germany, May 2000, pp. 265-268.
29. S. Morishima and H. Harashima, "A media conversion from speech to facial image for intelligent man-machine interface," IEEE J. Sel. Areas Commun., vol. 9, no. 4, pp. 594-600, May 1991. (Pubitemid 21645615)
30. R. Rao, T. Chen, and R. Mersereau, "Audio-to-visual conversion for multimedia communication," IEEE Trans. Industrial Electron., vol. 45, no. 1, pp. 15-22, Feb. 1998. (Pubitemid 128739734)
31. K. Choi, Y. Luo, and J. Hwang, "Hidden Markov model inversion for audio-to-visual conversion an MPEG-4 facial animation system," J. VLSI Signal Process., vol. 29, pp. 51-61, Aug. 2001. (Pubitemid 32693621)
32. G. Zoric, "Hybrid Approach to Real-Time Speech Driven Facial Gesturing of Virtual Characters," Ph.D. dissertation, Univ. of Zagreb, Zagreb, Croatia, Jul. 2010.
33. A. V. Nefian, L. Liang, X. Pi, X. Liu, and K. Murphy, "Dynamic Bayesian networks for audio-visual speech recognition," EURASIP J. Appl. Signal Process., vol. 2002, pp. 1274-1288, Jan. 2002.
34. J. Xue, J. Borgstrom, J. Jiang, L. Bernstein, and A. Alwan, "Acoustically-driven talking face synthesis using dynamic Bayesian networks," in Proc. IEEE Int. Conf. Multimedia and Expo (ICME 2006), Toronto, ON, Canada, Jul. 2006, pp. 1165-1168. (Pubitemid 46679928)
35. Y. Cao, W. Tien, P. Faloutsos, and F. Pighin, "Expressive speech-driven facial animation," ACM Trans. Graphics, vol. 24, pp. 1283-1302, Oct. 2005.
36. C. Busso, M. Bulut, C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. Chang, S. Lee, and S. Narayanan, "IEMOCAP: Interactive emotional dyadic motion capture database," J. Lang. Resources Eval., vol. 42, no. 4, pp. 335-359, Dec. 2008.
37. I. Pandzic and R. Forchheimer, MPEG-4 Facial Animation -The Standard, Implementations and Applications. New York: Wiley, Nov. 2002.
38. K. Arun, T. Huang, and S. Blostein, "Least-squares fitting of two 3-d point sets," IEEE Trans. Pattern Anal. Mach. Intell., vol. 9, no. 5, pp. 698-700, Sep. 1987.
39. N. Sarris, N. Grammalidis, and M. Strintzis, "FAP extraction using three-dimensional motion estimation," IEEE Trans. Circuits Syst. Video Technol., vol. 12, no. 8, pp. 865-876, Oct. 2002. (Pubitemid 35468411)
40. P. Boersma and D. Weeninck, "Praat, a system for doing phonetics by computer," Inst. of Phonetic Sci., Univ. of Amsterdam, Amsterdam, The Netherlands, Tech. Rep. 132, 1996 [Online]. Available: http://www.praat.org
41. F. Jensen and T. Nielsen, Bayesian Networks and Decision Graphs. New York, USA: Springer-Verlag, Nov. 2010.
42. Z. Ghahramani and M. I. Jordan, "Factorial hidden Markov models," Mach. Learn., vol. 29, pp. 245-273, Nov. 1997. (Pubitemid 127510040)
43. K. Murphy, "Dynamic Bayesian networks: Representation, inference and learning," Ph.D. dissertation, Univ. of California, Berkely, Fall, 2002.
44. D. Eberly, 3D Game Engine Design: A Practical Approach to Real-Time Computer Graphics. San Francisco, CA: Morgan Kaufmann, 2000.
45. K. Balci, "Xface: MPEG-4 based open source toolkit for 3D facial animation," in Proc. Conf. Adv. Vis. Interfaces (AVI 2004), Gallipoli, Italy, May 2004, pp. 399-402.
46. C. Dehon, P. Filzmoser, and C. Croux, "Robust methods for canonical correlation analysis," in Data Analysis, Classification, Related Methods. Berlin, Germany: Springer-Verlag, 2000, pp. 321-326.
47. C. Busso, S. Lee, and S. Narayanan, "Analysis of emotionally salient aspects of fundamental frequency for emotion detection," IEEE Trans. Audio, Speech, Lang. Process., vol. 17, no. 4, pp. 582-596, May 2009.