Can you handle this? The impact of object affordances on how co-speech gestures are produced

Language, Cognition and Neuroscience

Volumn 31, Issue 3, 2016, Pages 430-440

  Masson Carro, Ingrid ^a   Goudbeek, Martijn ^a   Krahmer, Emiel ^a  

^a TILBURG UNIVERSITY   (Netherlands)

Author keywords

action; affordances; Gesture; representation techniques; simulation

Indexed keywords

ARTICLE; CONVERSATION; EVOKED MUSCLE RESPONSE; GESTURE; HAND GESTURE; HUMAN; INFORMATION PROCESSING; INTERPERSONAL COMMUNICATION; LANGUAGE; NEUROIMAGING; PATTERN RECOGNITION; PERCEPTION; SIMULATION; SPEECH; TASK PERFORMANCE; VISIBILITY; VISUAL STIMULATION; VISUOMOTOR COORDINATION;

EID: 84961200631     PISSN: 23273798     EISSN: 23273801     Source Type: Journal    
DOI: 10.1080/23273798.2015.1108448     Document Type: Article

References (49)

1. D.J.Barr,, R.Levy,, C.Scheepers,, & H.J.Tily, (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278. doi: 10.1016/j.jml.2012.11.001
2. A.Bartolo,, R.Cubelli,, S.Della Sala,, & S.Drei, (2003). Pantomimes are special gestures which rely on working memory. Brain & Cognition, 53, 483–494. doi: 10.1016/S0278-2626(03)00209-4
3. J.Bavelas,, & S.Healing, (2013). Reconciling the effects of mutual visibility on gesturing: A review. Gesture, 13(1), 63–92. doi:10.1075/gest.13.1.03bav
4. J.B.Bavelas,, N.Chovil,, D.A.Lawrie,, & A.Wade, (1992). Interactive gestures. Discourse Processes, 15, 469–489. doi: 10.1080/01638539209544823
5. J.B.Bavelas,, J.Gerwing,, C.Sutton,, & D.Prevost, (2008). Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language, 58, 495–520. doi: 10.1016/j.jml.2007.02.004
6. G.Beattie,, & H.Shovelton, (2002). What properties of talk are associated with the generation of spontaneous iconic hand gestures?. British Journal of Psychology, 41, 403–417. doi: 10.1348/014466602760344287
7. K.Bergman,, & S.Kopp, (2009). Increasing expressiveness for virtual agents: Autonomous generation of speech and gesture. In K.Decker, J.Sichman, C.Sierra, & C.Castelfranchi (Eds.), Proceedings of the 8th international conference on autonomous agents and multiagent systems (pp. 361–368). Ann Arbor, MI: IFAAMAS.
8. D.N.Bub,, & M.E.J.Masson, (2006). Gestural knowledge evoked by objects as part of conceptual representations. Aphasiology, 20, 1112–1124. doi: 10.1080/02687030600741667
9. D.N.Bub,, M.E.J.Masson,, & C.M.Bukach, (2003). Gesturing and naming: The use of functional knowledge in object identification. Psychological Science, 14, 467–472. doi: 10.1111/1467-9280.02455
10. D.N.Bub,, M.E.J.Masson,, & G.S.Cree, (2008). Evocation of functional and volumetric gestural knowledge by objects and words. Cognition, 106, 27–58. doi: 10.1016/j.cognition.2006.12.010
11. M.Chu,, & S.Kita, (2008). Spontaneous gestures during mental rotation tasks: Insights into the microdevelopment of the motor strategy. Journal of Experimental Psychology: General, 137, 706–723. doi: 10.1037/a0013157
12. M.Chu,, & S.Kita, (2015). Co-thought and Co-speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning, Memory, and Cognition. Advance online publication. doi:10.1037/xlm0000168
13. S.W.Cook,, & M.K.Tanenhaus, (2009). Embodied communication: Speakers’ gestures affect listeners’ actions. Cognition, 113, 98–104. doi: 10.1016/j.cognition.2009.06.006
14. R.Ellis,, & M.Tucker, (2000). Micro-affordance: The potentiation of components of action by seen objects. British Journal of Psychology, 91(4), 451–471. doi:10.1348/000712600161934
15. P.Feyereisen,, & I.Havard, (1999). Mental imagery and production of hand gestures while speaking in younger and older adults. Journal of Nonverbal Behavior, 23, 153–171. doi: 10.1023/A:1021487510204
16. M.Fischer,, & R.Zwaan, (2008). Embodied language: A review of the role of the motor system in language comprehension. The Quarterly Journal of Experimental Psychology, 61(6), 825–850. doi:10.1080/17470210701623605
17. A.Galati,, & S.Brennan, (2014). Speakers adapt gestures to addressees’ knowledge: Implications for models of co-speech gesture. Language, Cognition and Neuroscience, 29(4), 435–451. doi:10.1080/01690965.2013.796397
18. C.Gerlach,, I.Law,, & O.B.Paulson, (2002). When action turns into words: Activation of motor-based knowledge during categorization of manipulable objects. Journal of Cognitive Neuroscience, 14, 1230–1239. doi: 10.1162/089892902760807221
19. J.Gerwing,, & J.Bavelas, (2004). Linguistic influences on gesture's form. Gesture, 4, 157–195. doi: 10.1075/gest.4.2.04ger
20. J.J.Gibson, (1986). The ecological approach to visual perception. New York: Psychology Press. doi:10.4324/9780203767764
21. A.M.Glenberg,, & M.P.Kaschak, (2002). Grounding language in action. Psychonomic Bulletin & Review, 9, 558–565. doi: 10.3758/BF03196313
22. A.M.Glenberg,, & D.A.Robertson, (2000). Symbol grounding and meaning: A comparison of high-dimensional and embodied theories of meaning. Journal of Memory and Language, 43(3), 379–401. doi:10.1006/jmla.2000.2714
23. S.Glover,, D.A.Rosenbaum,, J.Graham,, & P.Dixon, (2004). Grasping the meaning of words. Experimental Brain Research, 154, 103–108. doi: 10.1007/s00221-003-1659-2
24. S.Goldin-Meadow, (2003). Hearing gesture: How our hands help us think. Cambridge, MA: Harvard University Press.
25. U.Hadar,, & B.Butterworth, (1997). Iconic gestures, imagery and word retrieval in speech. Semiotica, 115, 147–72. doi: 10.1515/semi.1997.115.1-2.147
26. T.C.Handy,, S.T.Grafton,, N.M.Shroff,, S.Ketay,, & M.S.Gazzaniga, (2003). Graspable object grab attention when the potential for action is recognised. Nature Neuroscience, 6, 421–427. doi: 10.1038/nn1031
27. O.Hauk,, I.Johnsrude,, & F.Pulvermüller, (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41, 301–307. doi: 10.1016/S0896-6273(03)00838-9
28. M.Hoetjes,, R.Koolen,, M.Goudbeek,, E.Krahmer,, & M.Swerts, (2015). Reduction in gesture during the production of repeated references. Journal of Memory and Language, 79–80, 1–17. doi: 10.1016/j.jml.2014.10.004
29. A.B.Hostetter, (2014). Action attenuates the effect of visibility on gesture rates. Cognitive Science, 38(7), 1468–1481. doi:10.1111/cogs.12113
30. A.B.Hostetter,, & M.W.Alibali, (2008). Visible embodiment: Gestures as simulated action. Psychonomic Bulletin and Review, 15, 495–514. doi: 10.3758/PBR.15.3.495
31. A.B.Hostetter,, & M.W.Alibali, (2010). Language, gesture, action! A test of the gesture as simulated action framework. Journal of Memory and Language, 63, 245–257. doi: 10.1016/j.jml.2010.04.003
32. A.B.Hostetter,, M.W.Alibali,, & A.E.Bartholomew, (2011). Gesture during mental rotation. In L.Carlson, C.Hoelscher, & T.Shipley (Eds.), Proceedings of the 33rd annual meeting of the cognitive science society (pp. 1448–1454). Austin, TX: Cognitive Science Society.
33. T.F.Jaeger, (2008). Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models. Journal of Memory and Language, 59(4), 434–446. doi:10.1016/j.jml.2007.11.007
34. A.Kendon, (2004). Gesture. Visible action as utterance. Cambridge: Cambridge University Press. doi:10.1017/cbo9780511807572
35. R.M.Krauss,, Y.Chen,, & R.F.Gottesman, (2000). Lexical gestures and lexical access: A process model. In D.McNeill (Ed.), Language and gesture (pp. 261–283). New York, NY: Cambridge University Press. doi:10.1017/cbo9780511620850.017
36. S.J.Lederman,, & R.L.Klatzky, (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19, 342–368. doi: 10.1016/0010-0285(87)90008-9
37. M.M.Louwerse, (2011). Symbol interdependency in symbolic and embodied cognition. Topics in Cognitive Science, 3, 273–302. doi: 10.1111/j.1756-8765.2010.01106.x
38. M.M.Louwerse,, & P.Jeuniaux, (2010). The linguistic and embodied nature of conceptual processing. Cognition, 114, 96–104. doi: 10.1016/j.cognition.2009.09.002
39. D.McNeill, (1992). Hand and mind. What gestures reveal about thought. Chicago: University of Chicago Press.
40. C.Müller, (1998). Iconicity and gesture. In S.Santi, I.Guatiella, C.Cave, & G.Konopczyncki (Eds.), Oralité et Gestualité (pp. 321–328). Montreal: L'Harmattan.
41. F.Parrill, (2010). Viewpoint in speech-gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes, 25(5), 650–668. doi:10.1080/01690960903424248
42. P.Perniss,, & G.Vigliocco, (2014). The bridge of iconicity: From a world of experience to the experience of language. Philosophical Transactions of the Royal Society, 369(1651), 1–13. doi:10.1098/rstb.2014.0179
43. K.Pine,, D.Gurney,, & B.Fletcher, (2010). The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener. Journal of Nonverbal Behavior, 34(3), 169–178. doi:10.1007/s10919-010-0089-7
44. J.Streeck, (2008). Depicting by gesture. Gesture, 8(3), 285–301. doi:10.1075/gest.8.3.02str
45. J.Streeck, (2009). Gesturecraft: The manu-facture of meaning. Amsterdam: John Benjamins Publishing. doi:10.1017/s0047404513000262
46. M.Tettamanti,, G.Buccino,, M.C.Saccuman,, V.Gallese,, M.Danna,, P.Scifo,, … D.Perani, (2005). Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience, 17, 273–281. doi: 10.1162/0898929053124965
47. M.Tucker,, & R.Ellis, (1998). On the relations between seen objects and components of potential actions. Journal of Experimental Psychology: Human Perception and Performance, 24(3), 830. doi:10.1037//0096-1523.24.3.830
48. K.Van Nispen,, M.van de Sandt-Koenderman,, L.Mol,, & E.Krahmer, (2014). Pantomime Strategies: On regularities in how people translate mental representations into the gesture modality. In P.Bello, M.Guarini, M.McShane, & B.Scassellati (Eds.), Proceedings of the 36th annual conference of the cognitive science society (pp. 976–981). Austin, TX: Cognitive Science Society.
49. P.Wittenburg,, H.Brugman,, A.Russel,, A.Klassmann,, & H.Sloetjes, (2006). ELAN: a professional framework for multimodality research. In Proceedings of LREC, 5th international conference on language resources and evaluation. Paris: ELRA.