MULTIMODAL COMMUNICATION from MULTIMODAL THINKING - TOWARDS AN INTEGRATED MODEL of SPEECH and GESTURE PRODUCTION

International Journal of Semantic Computing

Volumn 2, Issue 1, 2008, Pages 115-136

  Kopp, Stefan ^a   Bergmann, Kirsten ^a   Wachsmuth, Ipke ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

Gesture; multimodal semantics; production model; speech

Indexed keywords

SPEECH; SPEECH COMMUNICATION;

AUTOMATIC PRODUCTION; COMPUTATIONAL MODEL; GESTURE; INTEGRATED ARCHITECTURE; INTEGRATED MODELING; MULTI-MODAL; MULTIMODAL COMMUNICATIONS; PRODUCTION MODELING;

SEMANTICS;

EID: 70449472708     PISSN: 1793351X     EISSN: 17937108     Source Type: Journal    
DOI: 10.1142/S1793351X08000361     Document Type: Article

References (50)

1. M. Alibali and S. Kita, The role of gesture in speaking and thinking: Insights from piagetian conservation tasks, in T. Sakamoto (ed.), Communicating Skills of Intention. Hitsuji Shobo, in press.
2. M. Alibali, S. Kita and A. J. Young, Gesture and the process of speech production: We think, therefore we gesture, Language and Cognitive Processes 15 (2000) 593-613.
3. G. L. Allen, Gestures accompanying verbal route directions: Do they point to a new avenue for examining spatial representations? Spatial Cognition and Computation 3 (2003) 259-268.
4. J. Bavelas, C. Kenwood, T. Johnson and B. Philips, An experimental study of when and how speakers use gestures to communicate, Gesture 2(1) (2002) 1-17.
5. K. Bergmann and S. Kopp, Verbal or visual: How information is distributed across speech and gesture in spatial dialog, in D. Schlangen and R. Fernandez (eds.), Proc. 10th Workshop on the Semantics and Pragmatics of Dialogue (2006), pp. 90-97.
6. J. Cassell, M. Stone and H. Yan, Coordination and context-dependence in the generation of embodied conversation, in Proceedings of the First International Conference on Natural Language Generation, 2000.
7. D. Croft and P. Thagard, Dynamic imagery: A computational model of motion and visual analogy, in L. Magnani and N. Nersessian (eds.), Model-Based Reasoning: Science, Technology, Values (Kluwer/Plenum, New York, 2002), pp. 259-274.
8. J. P. de Ruiter, The production of gesture and speech, in D. McNeill (ed.), Language and gesture (Cambridge University Press, 2000).
9. M. Denis, The description of routes: A cognitive approach to the production of spatial discourse, Current Psychology of Cognition 16 (1997) 409-458.
10. J. Glasgow, The imagery debated revisited: A computational perspective, Computational Intelligence 9 (4) (1993).
11. B. Hartmann, M. Mancini and C. Pelachaud, Implementing expressive gesture synthesis for embodied conversational agents, in S. Gibet et al. (eds.), Gesture in Human-Computer Interaction and Simulation 2005.
12. A. Herskovits, Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English (Cambridge University Press, 1986).
13. A. B. Hostetter and M. W. Alibali, Raise your hand if you're spatial-Relations between verbal and spatial skills and gesture production, Gesture 7(1) (2007) 73-95.
14. A. B. Hostetter, M. W. Alibali and S. Kita, Does sitting on your hands make you bite your tongue? The effects of gesture inhibition on speech during motor descriptions, in D. S. McNamara and J. G. Trafton (eds.), Proc. 29th Meeting of the Cognitive Science Society (Erlbaum, 2007), pp. 1097-1102.
15. A. B. Hostetter, M. W. Alibali and S. Kita, I see it in my hands' eye: Representational gestures reflect conceptual demands, Language Cogn. Processes 22 (2007) 313-336.
16. A. Kendon, Do gestures communicate? A review, Research on Language and Social Interaction 27(3) (1994) 175-200.
17. A. Kendon, Gesture-Visible Action as Utterance (Cambridge University Press, 2004).
18. D. Kieras, Beyond pictures and words: Alternative information-processing models for imagery effects in verbal memory, Psychological Bulletin 85 (1978) 532-554.
19. M. Kipp, M. Neff, K. H. Kipp and I. Albrecht, Towards natural gesture synthesis: Evaluating gesture units in a data-driven approach to gesture synthesis, in C. Pelachaud et al. (ed.), Proc. 7th Int. Conf. Intelligent Virtual Agents (2007).
20. S. Kita, How representational gestures helps speaking, in D. McNeill (ed.), Language and Gesture (Cambridge University Press, 2000), pp. 162-185.
21. S. Kita and A. Özyürek,What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking, Journal of Memory and Language 48 (2003) 16-32.
22. S. Kita, A. Özyürek, S. Allen, A. Brown, R. Furman and T. Ishizuka, Relations between syntactic encoding and co-speech gestures: Implications for a model of speech and gesture production, Language and Cognitive Processes (2007).
23. S. Kopp, P. Tepper and J. Cassell, Towards integrated microplanning of language and iconic gesture for multimodal output, in Proc. Int. Conf. Multimodal Interfaces (ACM Press, New York, 2004) pp. 97-104.
24. S. Kopp, P. Tepper, K. Ferriman, K. Striegnitz and J. Cassell, Trading spaces: How humans and humanoids use speech and gesture to give directions, in T. Nishida (ed.), Conversational Informatics (John Wiley, 2007), Chap. 8, pp. 133-160.
25. S. Kopp and I. Wachsmuth, Synthesizing multimodal utterances for conversational agents, Computer Animation and Virtual Worlds 15(1) (2004) 39-52.
26. S. M. Kosslyn, Seeing and imagining in the celebral hemispheres: A computational approach, Psychological Review 94 (1987) 148-175.
27. S. M. Kosslyn, Mental images and the brain, Cogn. Neuropsychology 22 (2005) 333-347.
28. R. M. Krauss, Y. Chen and R. F. Gottesman, Lexical gestures and lexical access: A process model, in D. McNeill (ed.), Language and Gesture (Cambridge University Press, 2000).
29. B. Landau and R. Jackendoff, "What" and "where" in spatial language and spatial cognition, Behavioral and Brain Sciences 16(2) (1993) 217-265.
30. E. Lang, The semantics of dimensional designation of spatial objects, in M. Bierwisch and E. Lang (eds.), Dimensional Adjectives: Grammatical Structure and Conceptual Interpretation (Springer, Berlin, 1989), pp. 263-417.
31. S. Levinson, Frames of reference and Molyneux's question: Cross-linguistic evidence, in P. Bloom et al. (eds.), Space and Language (MIT Press, 1996), pp. 109-169.
32. D. McNeill, Hand and Mind-What Gestures Reveal about Thought (University of Chicago Press, Chicago, 1992).
33. D. McNeill and S. Duncan, Growth points in thinking-for-speaking, in Language and Gesture (Cambridge University Press, 2000).
34. A. Melinger and W. J. M. Levelt, Gesture and the communicative intention of the speaker, Gesture 4 (2004) 119-141.
35. P. Morrel-Samuels and R. Krauss, Word familiarity predicts temporal asynchrony of hand gestures and speech, Journal of Experimental Psychology: Learning, Memory, &Cognition 18 (1992) 615-622.
36. E. Morsella and R. M. Krauss, The role of gestures in spatial working memory and speech, American Journal of Psychology 117 (2004) 411-424.
37. C. Müller, Redebegleitende Gesten: Kulturgeschichte-Theorie-Sprachvergleich (Berlin Verlag, 1998).
38. A. Özyürek, Speech-gesture synchrony in typologically different languages and second language acquisition, in Proc. Boston Univ. Conf. on Language Development (Cascadilla Press, 2002), pp. 500-509.
39. A. Özyürek, S. Kita, S. Allen, R. Furman, and A. Brown, How does linguistic framing influence co-speech gestures? Insights from crosslinguistic differences and similarities, Gesture 5 (2005) 216-241.
40. A. Paivio, Mental Representations (Oxford University Press, 1986).
41. F. H. Rauscher, R. M. Krauss and Y. Chen, Gesture, speech, and lexical access: The role of lexical movements in speech production, Psychological Science 7 (1996) 226-231.
42. E. Reiter and R. Dale, Building Natural Language Generation Systems (Cambridge University Press, 2000).
43. D. I. Slobin, From "thought and language" to "thinking for speaking", in J. J. Gumperz et al. (eds.), Rethinking Linguistic Relativity (Cambridge University Press, 1996), pp. 70-96.
44. T. Sowa, Understanding Coverbal Iconic Gestures in Shape Descriptions (Akademische Verlagsgesellschaft Aka, Berlin, 2006).
45. T. Sowa and I. Wachsmuth, A model for the representation and processing of shape in coverbal iconic gestures, in Proc. KogWis05 (Schwabe, Basel, 2005), pp. 183-188.
46. M. Stone, C. Doran, B. Webber, T. Bleam and M. Palmer, Microplanning with communicative intentions: The spud system, Comput. Intelligence 19 (2003) 311-381.
47. J. Streeck, Gesture: The Manufacture of Understanding (unpublished manuscript).
48. L. Talmy, Lexicalization patterns: Semantic structure in lexical forms, in T. Shopen (ed.), Language Typology and Syntactic Description (Cambridge University Press, 1985), Vol. 3: Grammatical Categories and the Lexicon, pp. 57-149.
49. N. J. T. Thomas, Mental imagery, in E. N. Zalta (ed.), The Stanford Encyclopedia of Philosophy (Fall 2007 Edition) (2007).
50. B. Tversky and P. U. Lee, How space structures language, in C. Freksa et al. (eds.), Spatial Cognition: An Interdisciplinary Approach to Representation and Processing of Spatial Knowledge (Springer, Berlin, 1998), pp. 157-176.