Predicting speaker head nods and the effects of affective information

IEEE Transactions on Multimedia

Volumn 12, Issue 6, 2010, Pages 552-562

  Lee, Jina ^a   Marsella, Stacy C ^a  

^a University of Southern California   (United States)

Author keywords

Embodied conversational agents; emotion; head nods; machine learning; nonverbal behaviors; virtual agents

Indexed keywords

EMBODIED CONVERSATIONAL AGENT; EMOTION; HEAD NODS; MACHINE-LEARNING; NONVERBAL BEHAVIOR; VIRTUAL AGENT;

FEATURE EXTRACTION; FORECASTING; HIDDEN MARKOV MODELS; LEARNING SYSTEMS;

MATHEMATICAL MODELS;

EID: 77956781864     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2010.2051874     Document Type: Article

References (40)

1. J. Lee and S. Marsella, "Nonverbal behavior generator for embodied conversational agents," in Proc. 6th Int. Conf. Intelligent Virtual Agents, 2006, pp. 243-255, Springer.
2. M. Knapp and J. Hall, Nonverbal Communication in Human Interaction, 4th ed. Fort Worth, TX: Harcourt Brace College, 1997.
3. J. K. Burgoon, , M. L. Knapp and G. R. Miller, Eds., "Nonverbal signals," in Handbook of Interpersonal Communication, 2nd ed. Thousand Oaks, CA: Sage, 1994, pp. 229-285.
4. L. Tickle-Degnen and R. Rosenthal, "The nature of rapport and its nonverbal correlates," Psychol. Inq., vol.1, no.4, pp. 285-293, 1990.
5. E. Z. McClave, "Linguistic functions of head movements in the context of speech," J. Pragmat., vol.32, pp. 855-878, 2000.
6. A. Kendon, "Some uses of the head shake," Gesture, vol.2, pp. 147-182, 2002.
7. D. Heylen, "Challenges ahead: Head movements and other social acts in conversations," in Proc. Social Presence Cues Symp (AISB 2005), 2005.
8. A. Mignault and A. Chaudhuri, "The many faces of a neutral face: Head tilt and perception of dominance and emotion," J. Nonverb. Behav., vol.2, no.27, pp. 111-132, Jun. 2003.
9. W. Breitfuss, H. Prendinger, and M. Ishizuka, "Automated generation of non-verbal behavior for virtual embodied characters," in Proc. 9th Int. Conf. Multimodal Interfaces (ICMI'07), New York, 2007, pp. 319-322, ACM.
10. J. Cassell, "More than just another pretty face: Embodied conversational interface agents," Commun. ACM, vol.43, pp. 70-78, 2000.
11. J. Cassell, H. H. Vilhjálmsson, and T. Bickmore, "BEAT: The behavior expression animation toolkit," in Proc. 28th Annu. Conf. Computer Graphics and Interactive Techniques (SIGGRAPH'01), New York, 2001, pp. 477-486, ACM.
12. M. E. Foster and J. Oberlander, "Corpus-based generation of head and eyebrow motion for an embodied conversational agent," Lang. Resour. Eval., vol.41, pp. 305-324, 2007.
13. F. L. A. Knoppel, A. S. Tigelaar, D. O. Bos, T. Alofs, and Z. Ruttkay, "Trackside DEIRA: A dynamic engaging intelligent reporter agent," in Proc. 7th Int. Joint Conf. Autonomous Agents and Multiagent Systems (AAMAS'08), Richland, SC, 2008, pp. 112-119, International Foundation for Autonomous Agents and Multiagent Systems.
14. M. Strauss and M. Kipp, L. Padgham, D. C. Parkes, J. Mller, and S. Parsons, Eds., "Eric: A generic rule-based framework for an affective embodied commentary agent," in Proc. 7th Int. Joint Conf. Autonomous Agents and Multiagent Systems (AAMAS'08), Richland, SC, 2008, pp. 97-104, International Foundation for Autonomous Agents and Multiagent Systems.
15. K. G. Munhall, J. A. Jones, D. E. Callan, T. Kuratate, and E. Vatikiotis-Bateson, "Visual prosody and speech intelligibility: Head movement improves auditory speech perception," Psychol. Sci., vol.15, pp. 133-137, Feb. 2004.
16. C. Busso, Z. Deng, M. Grimm, U. Neumann, and S. Narayanan, "Rigid head motion in expressive speech animation: Analysis and synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol.15, no.3, pp. 1075-1086, Mar. 2007.
17. W. Swartout, J. Gratch, R. W. Hill, E. Hovy, S. Marsella, J. Rickel, and D. Traum, "Toward virtual humans," AI Mag., vol.27, no.2, pp. 96-108, 2006.
18. D. Traum, A. Roque, A. L. P. Georgiou, J. Gerten, B. M. S. Narayanan, S. Robinson, and A. Vaswani, "Hassan: A virtual human for tactical questioning," in Proc. 8th SIGdial Workshop Discourse and Dialogue, Antwerp, Belgium, Sep. 2007, pp. 71-74, Association for Computational Linguistics.
19. P. G. Kenny, T. D. Parsons, J. Gratch, A. Leuski, and A. A. Rizzo, , C. Pelachaud, J.-C. Martin, E. André, G. Chollet, K. Karpouzis, and D. Pelé, Eds., "Virtual patients for clinical therapist skills training," in IVA, ser. Lecture Notes in Computer Science. New York: Springer, 2007, vol.4722, pp. 197-210.
20. R. W. Hill, J. Belanich, H. C. Lane, M. G. Core, M. Dixon, E. Forbell, J. Kim, and J. Hart, "Pedagogically structured game-based training: Development of the elect bilat simulation," in Proc. 25th Army Science Conf. (ASC 2006), Nov. 2006, Association for Computational Linguistics.
21. L. R. Rabiner, "A tutorial on hidden Markov models and selected applications in speech recognition," Proc. IEEE vol.77, no.2, pp. 257-286, Feb. 1989.
22. L.-P. Morency, I. de Kok, and J. Gratch, "Predicting listener backchannels: A probabilistic multimodal approach," in Proc. 8th Int. Conf. Intelligent Virtual Agents, 2008, pp. 176-190.
23. M. Kipp, M. Neff, K. H. Kipp, and I. Albrecht, "Towards natural gesture synthesis: Evaluating gesture units in a data-driven approach to gesture synthesis," in Proc. 7th Int. Conf. Intelligent Virtual Agents, 2007, pp. 15-28.
24. J. Lee and S. Marsella, "Learning a model of speaker head nods using gesture corpora," in Proc. 8th Int. Joint Conf. Autonomous Agents and Multiagent Systems (AAMAS'09), 2009, International Foundation for Autonomous Agents and Multiagent Systems.
25. J. Lee, A. Neviarouskaya, H. Prendinger, and S. Marsella, "Learning models of speaker head nods with affective information," in Proc. 3rd Int. Conf. Affective Computing and Intelligent Interaction (ACII'09), 2009, International Foundation for Autonomous Agents and Multiagent Systems.
26. G. Bente and N. Krämer, "Virtual gestures. Embodiment and nonverbal behavior in computer-mediated communication," in Emotion in the Internet, A. Kappas, Ed. Cambridge, U.K.: Cambridge Univ. Press, to be published.
27. B. Lance and S. Marsella, "Glances, glares, and glowering: Howshould a virtual human express emotion through gaze?," J. Autonom. Agents and Multi-Agent Syst., vol.20, no.1, pp. 50-69, 2010.
28. T. Ward and W. Tsukahara, "Visual prosody and speech intelligibility in English and Japanese," Pragmatics, vol.23, pp. 1177-1207, 2004.
29. R. Nishimura, N. Kitaoka, and S. Nakagawa, "A spoken dialog system for chat-like conversations considering response timing," TSD 2007, pp. 599-606.
30. N. Cathcart, J. Carletta, and E. Klein, "A shallowmodel of backchannel continuers in spoken dialogue," in Proc. 10th Conf. Eur. Chapter of the Association for Computational Linguistics (EACL '03), 2003, pp. 51-58.
31. R. M. Maatman, J. Gratch, and S. Marsella, "Natural behavior of a listening agent," in Proc. 5th Int. Conf. Intelligent Virtual Agents, 2005, pp. 25-36.
32. H. P. Graf, E. Cosatto, V. Strom, and F. J. Huang, "Visual prosody: Facial movements accompanying speech," in Proc. AFGR, 2002, pp. 381-386.
33. D. Matsumoto, "American-Japanese cultural differences in the recognition of universal facial expressions," J. Cross-Cultural Psychol., vol.23, pp. 72-84, 2002.
34. S. K. Maynard, "Interactional functions of a nonverbal sign: Head movement in Japanese dyadic casual conversation," J. Pragmat., vol.11, pp. 589-606, 1987.
35. J. Carletta, "Unleashing the killer corpus: Experiences in creating the multi-everything AMI meeting corpus," Lang. Resour. Eval. J., vol.41, no.2, pp. 181-190, 2007.
36. E. Charniak, "A maximum-entropy-inspired parser," in Proc. 1st Conf. North Amer. Chapter of the Association for Computational Linguistics, San Francisco, CA: Morgan Kaufmann, 2000, pp. 132-139.
37. A. D.Wilson and A. F. Bobick, "Parametric hidden Markov models for gesture recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol.21, no.9, pp. 884-900, Sep. 1999.
38. B. H. Hwang and L. R. Rabiner, "Hidden Markov models for speech recognition," Technometrics, vol.33, no.3, pp. 251-272, 1991.
39. E. Charniak, Statistical Language Learning. Cambridge, MA: MIT Press, 1993.
40. A. Neviarouskaya, H. Prendinger, and M. Ishizuka, "Textual affect sensing for sociable and expressive online communication," in Proc. 2nd Int. Conf. Affective Computing and Intelligent Interaction (ACII'07), Berlin, Germany: Springer-Verlag, 2007, pp. 218-229.