Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions

Conversational Informatics: An Engineering Approach

Volumn , Issue , 2007, Pages 133-160

  Kopp, Stefan ^a   Tepper, Paul A ^b   Ferriman, Kimberley ^c   Striegnitz, Kristina ^b   Cassell, Justine ^b  

^a BIELEFELD UNIVERSITY   (Germany)

^b NORTHWESTERN UNIVERSITY   (United States)

^c VANDERBILT UNIVERSITY   (United States)

Author keywords

Complex spatial layout illustration; Gesture speech combinations; Image description feature level (IDF); Inter rater reliability; Modality independent cognitive representation; Motion verbs and gesture manner; Perceptual and conceptual salience; Qualitative feature based approach; Single spatial structure; Visuo spatial meaning context independent level

Indexed keywords

EID: 44449132620     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9780470512470.ch8     Document Type: Chapter

References (53)

1. Black A. and Taylor P. (2002) The Festival Speech Synthesis System: System documentation. Retrieved. August 2007 from http://festvox.org/docs/manual-1.4.3.
2. Cassell J. and Prevost S. (1996) Distribution of semantic features across speech and gesture by humans and computers. Paper presented at the Workshop on the Integration of Gesture in Language and Speech, Newark, DE.
3. Cassell J., McNeill D. and McCullough K.-E. (1999) Speech-gesture mismatches: evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics and Cognition 7 (1), 1-33.
4. Cassell J., Stone M. and Yan H. (2000) Coordination and context-dependence in the generation of embodied conversation. Paper presented at the INLG 2000, Mitzpe Ramon, Israel.
5. Cassell J., Vilhjálmsson H. and Bickmore T. (2001) BEAT: the behavior expression animation toolkit. Paper presented at the SIGGRAPH 01, Los Angeles, CA.
6. Chi D., Costa M., Zhao L. and Badler N. (2000) The EMOTE model for effort and shape. Paper presented at the SIGGRAPH'00, New Orleans, LA.
7. Clark H.H. (1996) Using Language. Cambridge University Press, Cambridge.
8. Conklin E.J. and McDonald D.D. (1982) Salience: The key to the selection problem in natural language generation. Paper presented at the 20th Annual Meeting of the Association for Computational Linguistics, Toronto.
9. Couclelis H. (1996) Verbal directions for way-finding: space, cognition, and language. In J. Portugali (ed.), The Construction of Cognitive Maps. Kluwer Academic Publishers, Boston, pp. 133-153.
10. Daniel M.P., Heiser J. and Tversky B. (2003) Language and diagrams in assembly instructions. Paper presented at the European Workshop in Imagery and Cognition, Pavia, Italy.
11. Denis M. (1997) The description of routes: a cognitive approach to the production of spatial discourse. Current Psychology of Cognition 16, 409-458.
12. Emmorey K., Tversky B. and Taylor H.A. (2001) Using space to describe space: perspective in speech, sign, and gesture. Spatial Cognition and Computation 2 (3), 1-24.
13. Forbus K. (1983) Qualitative reasoning about space and motion. In D. Gentner and A.L. Stevens (eds), Mental Models. Erlbaum, Hillsdale, NJ, pp. 53-73.
14. Gao Y. (2002) Automatic Extraction of Spatial Location for Gesture Generation. MIT Press, Cambridge, MA.
15. Green N., Carenini G., Kerpedjiev S. and Roth S.F. (1998) A media-independent content language for integrated text and graphics generation. Paper presented at theWorkshop on Content Visualization and Intermedia Representations at COLING and ACL'98, University of Montreal, Quebec.
16. Guhe M., Habel C. and Tschander L. (2003) Incremental production of preverbal messages with in C. Paper presented at the 5th International Conference on Cognitive Modeling, Bamberg, Germany.
17. Heiser J. and Tversky B. (2003) Characterizing diagrams produced by individuals and dyads. Paper presented at the Workshop on Interactive Graphics, London.
18. Heiser J.L., Tversky B., Agrawala M. and Hanrahan P. (2003) Cognitive design principles for visualizations: Revealing and instantiating. Paper presented at the Cognitive Science Society Meetings.
19. Heiser J., Phan D., Agrawala M., Tversky B. and Hanrahan P. (2004) Identification and validation of cognitive design principles for automated generation of assembly instructions. Paper presented at the Advanced Visual Interfaces'04.
20. Herskovits A. (1986) Language and Spatial Cognition: An interdisciplinary study of prepositions in English. Cambridge University Press, Cambridge.
21. Jackendoff R. and Landau B. (1991) Spatial language and spatial cognition. In D.J. Napoli, J.A. Kegl et al. (eds), Bridges between Psychology and Linguistics: A Swarthmore Festschrift for Lila Gleitman. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 145-169.
22. Joshi A. (1987) The relevance of tree adjoining grammar to generation. In G. Kempen (ed.), Natual Language Generation: new results in artificial and intelligence, psychology and linguistics. Kluwer Academic Publishers, Boston, pp. 233-252.
23. Kerpedjiev S., Carenini G., Green N., Moore J. and Roth S. (1998) Saying It In Graphics: from intentions to visualizations. Proceedings of the IEEE Symposium on Information Visualization, pp. 97-101.
24. Kopp S. andWachsmuth I. (2004) Synthesizing multimodal utterances for conversational agents. Journal of Computer Animation and Virtual Worlds 15 (1), 39-52.
25. Kopp S., Tepper P. and Cassell J. (2004) Towards integrated microplanning of language and iconic gesture for multimodal output. Proceedings of International Conference on Multimodal Interfaces (ICMI), ACM Press, pp. 97-104.
26. Krauss R.M., Morrel-Samuels P. and Colasante C. (1991) Do conversational hand gestures communicate? Journal of Personality and Social Psychology 61 (5), 743-754.
27. Landau B. and Jackendoff R. (1993) "What" and "where" in spatial language and spatial cognition. Behavioral and Brain Sciences 16 (2), 217-265.
28. Lozano S. and Tversky B. (2004) Communicative gestures benefit communicators. Paper presented at CogSci 2004, Chicago, IL.
29. McNeill D. (1992) Hand and Mind: What gestures reveal about thought. University of Chicago Press, Chicago.
30. McNeill D. and Levy E. (1982) Conceptual representations in language activity and gesture. Speech, Place, and Action, 271-295.
31. Milde J.-T. and Gut U. (2002) The TASX-environment: an XML-based toolset for time aligned speech corpora. Paper presented at the 3rd International Conference on Language Resources and Evaluation (LREC), Las Palmas.
32. Nijholt A., Theune M. and Heylen D. (2005) Embodied language generation. In O. Stock and M. Zancanaro (eds), Intelligent Information Presentation, Springer, vol. 27, pp. 47-70.
33. Peirce C. (1955) Philosophical Writings of Peirce. Dover Publications, New York.
34. Perlin K. and Goldberg A. (1996) Improv: a system for scripting interactive actors in virtual worlds. Paper presented at the SIGGRAPH'96, New Orleans, LA.
35. Poesio M. (1996) Semantic ambiguity and perceived ambiguity. In K. van Deemter and S. Peters (eds), Ambiguity and Underspecification. CSLI, Palo Alto.
36. Poesio M. (2005) Incrementality and Underspecification in Semantic Processing. CSLI, Palo Alto.
37. Prillwitz S. (1989) HamNoSys: Hamburg Notation System for Sign Languages: an introductory guide (ver. 2.0 ed., Vol. 5). Signum Press, Hamburg.
38. Reiter E. and Dale R. (2000) Building Natural Language Generation Systems. Cambridge University Press, Cambridge.
39. Rickel J., Marsella S., Gratch J., Hill R., Traum D. and Swartout W. (2002) Toward a new generation of virtual humans for interactive experiences. IEEE Intelligent Systems 17 (4), 32-38.
40. Saussure F.d. (1985) Cours de linguistique générale. Payot, Paris.
41. Sowa T. and Wachsmuth I. (2003) Coverbal iconic gestures for objects descriptions in virtual environments: an empirical study. Paper presented at the Gestures, Meaning and Use Conference, Edicoes Fernando Pessoa.
42. Stone M., Doran C., Webber B., Bleam T. and Palmer M. (2003) Microplanning with communicative intentions: the SPUD system. Computational Intelligence 19 (4), 311-381.
43. Striegnitz K., Tepper P., Lovett A. and Cassell J. (to appear). Knowledge representation for generating locating gestures in route directions. In K. Coventry, T. Tenbrink and J. Bateman (eds), Spatial Language in Dialogue. Oxford University Press.
44. Talmy L., University of California Berkeley, Institute of Cognitive Studies (1983). How Language Structures Space. Cognitive Science Program, Institute of Cognitive Studies, University of California at Berkeley.
45. Talmy L. (2000) Toward a Cognitive Semantics. MIT Press, Cambridge, MA.
46. Taylor H.A. and Tversky B. (1992) Spatial mental models derived from survey and route descriptions. Journal of Memory and Language 31, 261-282.
47. Taylor H.A. and Tversky B. (1996) Perspective in spatial descriptions. Journal of Memory and Language 35, 371-391.
48. Towns S., Callaway C. and Lester J. (1998) Generating coordinated natural language and 3D animations for complex spatial explanations. Paper presented at the AAAI-98.
49. Traum D. and Rickel J. (2002) Embodied agents for multi-party dialogue in immersive virtual worlds. Paper presented at the Autonomous Agents and Multi-Agent Systems, Melbourne.
50. Tverskey B. and Lee P. (1998) How space structures language. In C. Freksa, C. Habel and K.F. Wender (Eds), Spatial Cognition: An Interdisciplinary Approach to Representation and Processing of Spatial Knowledge. Springer, Berlin, 57-75.
51. Tverskey B. and Lee P. (1999) Pictorial and verbal tools for conveying routes. In C. Freksa and D. Mark (Eds), Spatial Information Theory: Cognitive and Computational Foundations of Geographic Information Science. Springer, Berlin, 51-64.
52. Yan H. (2000) Paired Speech and Gesture Generation in Embodied Conversational Agents. Unpublished Masters of Science thesis, MIT, Cambridge, MA.
53. Young R.M. and Moore J.D. (1994) DPOCL: a principled approach to discourse planning. Paper presented at the 7th International Workshop on Natural Language Generation, Kennebunkport, ME.