The Interplay Between Gesture and Speech in the Production of Referring Expressions: Investigating the Tradeoff Hypothesis

Topics in Cognitive Science

Volumn 4, Issue 2, 2012, Pages 232-248

  de Ruiter, Jan P ^a,b   Bangerter, Adrian ^c   Dings, Paula ^b  

^a BIELEFELD UNIVERSITY   (Germany)

^b MAX PLANCK INSTITUTE FOR PSYCHOLINGUISTICS   (Netherlands)

^c UNIVERSITY OF NEUCHÂTEL   (Switzerland)

Author keywords

Gesture; Gesture speech redundancy; Gesture speech tradeoff; Iconic gestures; Pointing; Referring expressions; Speech production

Indexed keywords

ARTICLE; GESTURE; HUMAN; LANGUAGE; NONVERBAL COMMUNICATION; OBSERVER VARIATION; PHYSIOLOGY; SPEECH;

GESTURES; HUMANS; LANGUAGE; MANUAL COMMUNICATION; OBSERVER VARIATION; SPEECH;

EID: 84859700682     PISSN: 17568757     EISSN: 17568765     Source Type: Journal    
DOI: 10.1111/j.1756-8765.2012.01183.x     Document Type: Article

References (57)

1. Alibali, M. W., Heath, D. C., & Myers, H. J. (2001). Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language, 44, 169-188.
2. Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59, 390-412.
3. Bangerter, A. (2004). Using pointing and describing to achieve joint focus of attention in dialogue. Psychological Science, 15, 415-419.
4. Bard, E. G., Anderson, A. H., Chen, Y., Nicholson, H., Havard, C., & Dalzel-Job, S. (2007). Let's you do that: Sharing the cognitive burdens of dialogue. Journal of Memory and Language, 57, 616-641.
5. Bavelas, J. (1994). Gestures as part of speech: Methodological implications. Research in Language and Social Interaction, 27, 201-221.
6. Bavelas, J., Gerwing, J., Sutton, C., & Prevost, D. (2008). Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58, 495-520.
7. Bergmann, K., & Kopp, S. (2009). GNetic-Using Bayesian decision networks for iconic gesture generation. Paper presented at the IVA09, Amsterdam, The Netherlands.
8. Boersma, P., & Weenink, D. (2007). Praat: Doing phonetics by computer (version 4.2). Available at:
9. Brennan, S. E., & Clark, H. H. (1996). Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22, 1482-1493.
10. Brugman, H., & Russel, A. (2004). Annotating multimedia/multi-modal resources with ELAN. In Proceedings of LREC 2004, fourth international conference on language resources and evaluation, Lisbon, Portugal.
11. Clark, H. H. (1996). Using language. Cambridge, MA: Cambridge University Press.
12. Clark, H. H., & Krych, M. A. (2004). Speaking while monitoring addressees for understanding. Journal of Memory and Language, 50, 62-81.
13. Clark, H. H., & Murphy, G. L. (1982). Audience design in meaning and reference. In J. F. Leny & W. Kintsch (Eds.), Language and comprehension (pp. 287-299). Amsterdam, The Netherlands: North Holland.
14. Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22, 1-39.
15. De Ruiter, J. P. (1998). Gesture and speech production. Unpublished Doctoral Dissertation, Radboud University, Nijmegen.
16. De Ruiter, J. P. (2000). The production of gesture and speech. In D. McNeill (Ed.), Language and gesture (pp. 284-311). Cambridge, UK: Cambridge University Press.
17. De Ruiter, J. P. (2006). Can gesticulation help aphasic people speak, or rather, communicate? Advances in Speech Language Pathology, 8, 124-127.
18. De Ruiter, J. P. (2007). Postcards from the mind: The relationship between thought, imagistic gesture, and speech. Gesture, 7, 21-38.
19. Engelhardt, P. E., Bailey, K. G. D., & Ferreira, F. (2006). Do speakers and listeners observe the Gricean Maxim of Quantity? Journal of Memory and Language, 54, 554-573.
20. Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391.
21. Gerwing, J., & Bavelas, J. B. (2004). Linguistic influences on gesture's form. Gesture, 4, 157-195.
22. Holler, J., & Stevens, R. (2007). The effect of common ground on how speakers use gesture. Journal of Language and Social Psychology, 26, 4-27.
23. Holler, J., & Wilkin, K. (2009). Communicating common ground: How mutually shared knowledge influences speech and gesture in a narrative task. Language and Cognitive Processes, 24, 267-289.
24. Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review, 15, 495-514.
25. Hupet, M., Seron, X., & Chantraine, Y. (1991). The effect of the codability and discriminability of the referents on the collaborative referring procedure. British Journal of Psychology, 82, 449-462.
26. Jacobs, N., & Garnham, A. (2007). The role of conversational hand gestures in a narrative task. Journal of Memory and Language, 56, 291-303.
27. Jescheniak, J. D., & Levelt, W. J. M. (1994). Word frequency effects in speech production: Retrieval of syntactic information and of phonological form. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 824-843.
28. Kempen, G., & Huijbers, P. (1983). The lexicalization process in sentence production and naming: Indirect election of words. Cognition, 14, 185-209.
29. Kendon, A. (1972). Some relationships between body motion and speech. In A. W. Sigman & B. Pope (Eds.), Studies in dyadic communication (pp. 177-216). New York: Pergamon Press.
30. Kendon, A. (1994). Do gestures communicate?: A review. Research in Language and Social Interaction, 27, 175-200.
31. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
32. Keysar, B., Barr, D. J., & Balin, J. A. (1998). Definite reference and mutual knowledge: Process models of common ground in comprehension. Journal of Memory and Language, 39, 1-20.
33. Keysar, B., Barr, D. J., Balin, J. A., & Brauner, J. S. (2000). Taking perspective in conversation. The role of mutual knowledge in comprehension. Psychological Science, 11, 32-38.
34. Kita, S. (2000). How representational gestures help speaking. In D. McNeill (Ed.), Language and gesture: Window into thought and action (pp. 162-185). Cambridge, UK: Cambridge University Press.
35. Kranstedt, A., Kopp, S., & Wachsmuth, I. (2002). MURML: A multimodal utterance representation markup language for conversational agents. AAMAS'02 Workshop on Embodied Conversational Agents, Bologna, Italy.
36. Krauss, R. M. (1998). Why do we gesture when we speak? Current Directions in Psychological Science, 7, 54-60.
37. Krauss, R. M., Apple, W., Morency, N., Wenzel, C., & Winton, W. (1981). Verbal, vocal, and visible factors in judgments of another's affect. Journal of Personality and Social Psychology, 40, 312-319.
38. Krauss, R. M., Chen, Y., & Chawla, P. (1995). Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us? In M. Zanna (Ed.), Advances in Experimental Social Psychology (Vol. 28, pp. 389-450). Tampa, FL: Academic Press.
39. Krauss, R. M., Chen, Y., & Gottesmann, R. F. (2000). Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture (pp. 261-283). Cambridge, UK: Cambridge University Press.
40. Krauss, R. M., Morrel-Samuels, P., & Colasante, C. (1991). Do conversational hand gestures communicate? Journal of Personality & Social Psychology, 61, 743-754.
41. Krauss, R. M., & Weinheimer, S. (1966). Concurrent feedback, confirmation and the encoding of referents in verbal communication. Journal of Personality and Social Psychology, 4, 343-346.
42. Levelt, W. J. M. (1989). Speaking: From intention to articulation. Cambridge, MA: The MIT Press.
43. Locker, L., Hoffman, L., & Bovaird, J. A. (2007). On the use of multilevel modeling as an alternative to items analysis in psycholinguistic research. Behavior Research Methods, 39, 723-730.
44. McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92, 350-371.
45. McNeill, D. (1992). Hand and mind. Chicago: The Chicago University Press.
46. McNeill, D., & Duncan, S. (2000). Growth points in thinking-for-speaking. In D. McNeill (Ed.), Language and gesture (pp. 141-161). Cambridge, UK: Cambridge University Press.
47. Melinger, A., & Kita, S. (2007). Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22, 473-500.
48. Melinger, A., & Levelt, W. J. M. (2004). Gesture and the communicative intention of the speaker. Gesture, 4, 119-141.
49. Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial working memory and speech. American Journal of Psychology, 117, 411-424.
50. Roelofs, A. (1992). A spreading activation theory of lemma retrieval in speaking. Cognition, 4, 2.
51. Schegloff, E. A. (1972). Notes on a conversational practice: Formulating place. In D. N. Sudnow (Ed.), Studies in social interaction (pp. 75-119). New York: McMillan, The Free Press.
52. Schriefers, H., De Ruiter, J. P., & Steigerwald, M. (1999). Parallelism in the production of noun phrases: Experiments and reaction time models. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 702-720.
53. So, W. C., Kita, S., & Goldin-Meadow, S. (2009). Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand. Cognitive Science, 33, 115-125.
54. Van der Sluis, I. (2005). Multimodal reference. Unpublished Doctoral Dissertation, Katholieke Universiteit Brabant, Tilburg.
55. Van der Sluis, I., & Krahmer, E. (2004). The influence of target size and distance on the production of speech and gesture in multimodal referring expressions. Paper presented at the The 8th International Conference on Spoken Language Processing (ICSLP), Jeju Island, Korea.
56. Van der Sluis, I., & Krahmer, E. (2007). Generating multimodal references. Discourse Processes, 44, 145-174.
57. Wesp, R. K., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. American Journal of Psychology, 114, 591-600.