Towards integrated microplanning of language and iconic gesture for multimodal output

ICMI'04 - Sixth International Conference on Multimodal Interfaces

Volumn , Issue , 2004, Pages 97-104

  Kopp, Stefan ^a   Tepper, Paul ^b   Cassell, Justine ^b  

^a BIELEFELD UNIVERSITY   (Germany)

^b NORTHWESTERN UNIVERSITY   (United States)

Author keywords

Embodied conversational agents; Generation; Gesture; Language; Multimodal output

Indexed keywords

GRAPHIC METHODS; IMAGE PROCESSING; INFORMATION ANALYSIS; LINGUISTICS; MATHEMATICAL MODELS; NATURAL LANGUAGE PROCESSING SYSTEMS; PATTERN RECOGNITION; SEMANTICS;

EMBODIED CONVERSATIONAL AGENTS; GENERATION; GESTURES; MULTIMODAL OUTPUT; NATURAL LANGUAGE;

GESTURE RECOGNITION;

EID: 14944386410     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1027933.1027952     Document Type: Conference Paper

References (29)

1. Cassell, J. & Prevost, S. Distribution of Semantic Features Across Speech and Gesture by Humans and Computers. In Proc. Workshop on Integration of Gesture in Language and Speech, 1996, Wilmington, DE.
2. Cassell, J., McNeill, D. & McCullough, K.E. Speech-Gesture Mismatches: Evidence for One Underlying Representation of Linguistic and Non-Linguistic Information. Pragmatics and Cognition 7(1): 1-33, 1999.
3. Cassell, J., Stone, M. & Yan, H. Coordination and context-dependence in the generation of embodied conversation. In Proc. INLG 2000. Mitzpe Ramon, Israel.
4. Cassell, J., Vilhjalmsson, H. & Bickmore, T. BEAT: the behavior expression animation toolkit. In Proc. SIGGRAPH 2001, pp. 477-486.
5. Chi, D., Costa, M., Zhao, L. & Badler, N. The EMOTE model for effort and shape. In Proc. SIGGRAPH 2000, pp. 173-182.
6. Clark, H. H. Using Language. Cambridge Univ. Press, 1996.
7. de Ruiter, J.P. The production of gesture and speech. In McNeill, D. (ed.) Language and Gesture. Cambridge, UK: Cambridge University Press, 2000.
8. Emmorey, K., Tversky, B. & Taylor, H.A. Using space to describe space: Perspective in speech, sign, and gesture. Spatial Cognition and Computation 2:3, pp. 157-180, 2000.
9. Gao, Y. Automatic extraction of spatial location for gesture generation, Master thesis, MIT Dept. of Electrical Engineering and Computer Science, 2002.
10. Green, N., G. Carenini, et al. A Media-Independent Content Language for Integrated Text and Graphics Generation. Workshop on Content Visualization and Intermedia Representations at COLING/ACL '98, Montreal, 1998.
11. Herskovits, A. Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English. Cambridge University Press, 1986.
12. Joshi, A.K. An Introduction to Tree Adjoining Grammars, In A. Manaster-Ramer (ed), Mathematics of Language, Amsterdam: John Benjamins, pp. 87-114, 1987.
13. Kerpedjiev, S., Carenini, G., Roth, S. & Moore, J. D. AutoBrief: A multimedia presentation system for assisting data analysis. In Computer Standards and Interfaces, 18: 583-593, 1997.
14. Koons, D.B., Sparrell, C.J., & Thorisson, K. Integrating simultaneous input from speech, gaze, and hand gestures. In M. Maybury (ed), Intelligent Multimedia Interfaces, pp. 252-276. Menlo Park, CA: MIT Press, 1993.
15. Kopp, S. & Wachsmuth, I. Synthesizing Multimodal Utterances for Conversational Agents. Computer Animation and Virtual Worlds: 15(1), pp. 39-52, 2004.
16. Landau, B. & Jackendoff, R. What and where in spatial language and spatial cognition. Behavioral and Brain Sciences 16: 217-265, 1993.
17. McNeill, D. & Levy, E. Conceptual representations in language activity and gesture. In R. Jarvella, & W. Klein (eds.): Speech, Place, and Action, John Wiley & Sons, 1982.
18. McNeill, D., Hand and Mind: What Gestures Reveal About Thought. Chicago, IL: Univ. of Chicago Press, 1992.
19. McNeill, D. Catchments and Contexts: Non-modular factors in speech and gesture production. In D. McNeill (ed.): Language and Gesture. Cambridge University Press, 2000.
20. Nijholt, A., Theune, M. & Heylen, D. Embodied Language Generation, In O. Stock & M. Zancanaro (eds): Intelligent Information Presentation, Kluwer, 2004.
21. Perlin, K. & Goldberg, A. Improv: A System for Scripting Interactive Actors in Virtual Worlds. In Proc. SIGGRAPH '96, pp. 205-216.
22. Pelachaud, C. & Poggi, I. Multimodal Embodied Agents, In Autonomous Agents Workshop Multimodal Communication and Context in Embodied Agents, pp. 95-99, 2001.
23. Reiter, E. & Dale, R. Building Natural Language Generation Systems. Cambridge, UK: Cambridge University Press, 2000.
24. Rickel, J., Marsella, S., Gratch, J., Hill, R., Traum, D. & Swartout, W. Toward a New Generation of Virtual Humans for Interactive Experiences. IEEE Intelligent Systems 17(4): 32-38, 2002.
25. Sowa, T. & Wachsmuth, I. Coverbal Iconic Gestures for Object Descriptions in Virtual Environments: An Empirical Study. In M. Rector, I. Poggi & N. Trigo (eds.): Proc. "Gestures. Meaning and Use", pp. 365-376, 2003.
26. Stone, M., Doran, C., Webber, B., Bleam, T. & Palmer, M. Microplanning with communicative intentions: the SPUD system. Computational Intelligence 19(4): 311-381, 2003.
27. Towns, S., Callaway, C. & Lester, J. Generating Coordinated Natural Language and 3D Animations for Complex Spatial Explanations. In Proc. AAAI-98, pp. 112-119, 1998.
28. Traum, D. & Rickel, J. Embodied Agents for Multi-party Dialogue in Immersive Virtual Worlds. In Proc. Autonomous Agents and Multi-Agent Systems, pp. 766-773, 2002.
29. Van, H. Paired Speech and Gesture Generation in Embodied Conversational Agents. Masters Thesis. MIT, School of Architecture and Planning, 2000.