Are depictive gestures like pictures? Commonalities and differences in semantic processing

Brain and Language

Volumn 119, Issue 3, 2011, Pages 184-195

  Wu, Ying Choon ^a,b   Coulson, Seana ^a,c  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Action comprehension; Gesture; Gesture N450; N300; N400; Object recognition; Semantic processing

Indexed keywords

ARTICLE; BODY MOVEMENT; BRAIN ELECTROPHYSIOLOGY; BRAIN FUNCTION; BRAIN MAPPING; COGNITION; ELECTROENCEPHALOGRAM; EVENT RELATED POTENTIAL; EYE MOVEMENT; GESTURE; HUMAN; HUMAN EXPERIMENT; IMAGE PROCESSING; IMAGING; NORMAL HUMAN; PATTERN RECOGNITION; SEMANTICS; STIMULUS RESPONSE; TASK PERFORMANCE; VIDEORECORDING; VISION; VISUAL ADAPTATION; VISUAL MEMORY; WAVEFORM;

BRAIN; COMPREHENSION; ELECTROENCEPHALOGRAPHY; EVOKED POTENTIALS; GESTURES; HUMANS; SEMANTICS; VISUAL PERCEPTION;

EID: 80054103162     PISSN: 0093934X     EISSN: 10902155     Source Type: Journal    
DOI: 10.1016/j.bandl.2011.07.002     Document Type: Article

References (67)

1. Allison T., Puce A., McCarthy G. Social perception from visual cues: Role of the STS. Trends in Cognitive Sciences 2000, 4(7):267-278.
2. Barrett S.E., Rugg M.D. Event-related potentials and the semantic matching of pictures. Brain and Cognition 1990, 14:201-212.
3. Bracci S., Ietswaart M., Peelen M.V., Cavina-Pratesi C. Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex. Journal of Neurophysiology 2010, 103(6):3389-3397.
4. Buccino G., Binkofski F., Riggio L. The mirror neuron system and action recognition. Brain and Language 2004, 89:370-376.
5. Corballis M.C. From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences 2003, 26:199-260.
6. Critescu T.C., Nobre A.C. Differential modulation of word recognition by semantic and spatial orienting of attention. Journal of Cognitive Neuroscience 2008, 20(5):787-801.
7. Downing P.E., Chan A.W., Peelen M.V., Dodds C.M., Kanwisher N. Domain specificity in visual cortex. Cerebral Cortex 2006, 16(10):1453-1461.
8. Federmeier K.D., Kutas M. Meaning and modality: Influences of context, semantic memory organization, and perceptual predictability on picture processing. Journal of Experimental Psychology: Learning, Memory, and Cognition 2001, 27:202-224.
9. Federmeier K.D., Kutas M. Picture the difference: Eletrophysiological investigations of picture processing in the two cerebral hemispheres. Neuropsychologia 2002, 40:730-747.
10. Feyereisen P., de Lannoy J.D. Gestures and speech: Psychological investigations 1991, Cambridge University Press, New York, NY.
11. Ganis G., Kutas M., Sereno M.I. The search for " common sense": An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience 1996, 8(2):89-106.
12. Geisser S., Greenhouse S. On methods in the analysis of profile data. Psychometrika 1959, 24:95-112.
13. Graham J.A., Argyle M. A cross-cultural study of the communication of extra-verbal meaning by gestures. International Journal of Psychology 1975, 10:57-67.
14. Grill-Spector K., Kourtzi Z., Kanwisher N. The lateral occipital complex and its role in object recognition. Vision Research 2001, 41:1409-1422.
15. Gunter T.C., Bach P. Communicating hands: ERPs elicited by meaningful symbolic hand postures. Neuroscience Letters 2004, 372:52-56.
16. Hamm J.P., Johnson B.W., Kirk I.J. Comparison of the N300 and N400 ERPs to picture stimuli in congruent and incongruent contexts. Clinical Neurophysiology 2002, 113:1339-1450.
17. Holcomb P.J. Automatic and attentional processing: An event-related brain potential analysis of semantic processing. Brain and Language 1988, 35:66-85.
18. Holcomb P.J., Kounios J., Anderson J.E., West C.W. Dual-coding, context-availability, and concreteness effects in sentence comprehension: An electrophysiological investigation. Journal of Experimental Psychology: Learning, Memory, and Cognition 1999, 25(3):721-742.
19. Holcomb P.J., McPherson W.B. Event-related brain potentials reflect semantic priming in an object decision task. Brain and Cognition 1994, 24:259-276.
20. Holle H., Gunter T.C., Ruschemeyer S.-A., Henenlotter A., Iacoboni M. Neural correlates of the processing of co-speech gestures. NeuroImage 2008, 15(39):2010-2024.
21. Iacoboni M. Imitation, empathy, and mirror neurons. Annual Review of Psychology 2009, 60:653-670.
22. Kelly S.D., Creigh P., Bartolotti J. Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience 2009, 22(4):683-694.
23. Kelly S.D., Manning S.M., Rodak S. Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology, and education. Language and Linguistics Compass 2008, 2(4):569-588.
24. Kelly S.D., Ozyurek A. Gesture, brain, and language. Brain and Language 2007, 101(3):181-184.
25. Kelly S.D., Ward S., Creigh P., Bartolotti J. An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language 2007, 101:222-233.
26. Kendon A. Some relationships between body motion and speech. Studies in dyadic communication 1972, 177-210. Pergamon Press, New York. A. Siegman, B. Pope (Eds.).
27. Kendon A. Gesture: Visible action as utterance 2004, Cambridge University Press, Cambridge.
28. Kounios J., Holcomb P.J. Concreteness effects in semantic processing: ERP evidence supporting dual-coding theory. Journal of Experimental Psychology: Learning, Memory and Cognition 1994, 20(4):804-823.
29. Kourtzi Z., Kanwisher N. Activation in human MT/MST by static images with implied motion. Journal of Cognitive Neuroscience 2000, 12(1):48-55.
30. Kutas M., Federmeier K.D., Sereno M.I. Current approaches to mapping language in electromagnetic space. The neurocognition of language 1999, 359-392. Oxford University Press. C.M. Brown, P. Hagoort (Eds.).
31. Kutas M., Hillyard S.A. Event-related potentials to semantically inappropriate and surprisingly large words. Biological Psychology 1980, 11:99-116.
32. Kutas M., Hillyard S.A. Brain potentials during reading reflect word expectancy and semantic association. Nature 1984, 307:161-163.
33. Lim V.K., Wilson A.J., Hamm J.P., Phillips N., Iwabuchi S.J., Corballis M.C., et al. Semantic processing of mathematical gestures. Brain and Cognition 2009, 71:306-312.
34. Liu J., Harris A., Kanwisher N. Perception of face parts and face configurations: An fMRI study. Journal of Cognitive Neuroscience 2009, 22(1):203-211.
35. McNeil N., Alibali M.W., Evans J.L. The role of gesture in children's comprehension of spoken language: Now they need it, now they don't. Journal of Nonverbal Behavior 2000, 24(2):131-150.
36. McNeill D. Hand and mind 1992, Chicago University Press, Chicago.
37. McPherson W.B., Holcomb P.J. An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology 1999, 36:53-65.
38. Mudrik L., Lamy D., Deouell L.Y. ERP evidence for context congruity effects during simultaneous object-scene processing. Neuropsychologia 2010, 48:507-517.
39. Nunez R.E., Sweetser E. With the future behind them: Convergent evidence from Aymara language and gesture in crosslinguistic comparison of spatial construals of time. Cognitive Science 2006, 30(3):401-450.
40. Nuwer M.R., Comi G., Emerson R., Fuglsang-Frederiksen A., Guerit J.M., Hinrichs H., et al. IFCN standards for digital recording of clinical EEG. The International Federation of Clinical Neurophysiology. Electroencephalography and Clinical Neurophysiology Supplement 1999, 52(11-14).
41. Obermeier, C., Dolk, T., & Gunter, T. C. (2011). The benefit of gestures during communication: Evidence from hearing and hearing-impaired individuals. Cortex.
42. Oldfield R.C. The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 1971, 9:97-113.
43. Op de Beeck H.P., Brants M., Baeck A., Wagemans J. Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex. NeuroImage 2010, 49(4):3414-3425.
44. Ozyurek A., Willems R.M., Kita S., Hagoort P. On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience 2007, 19(4):605-616.
45. Paulmann S., Jessen S., Kotz S.A. Investigating the multimodal nature of human communication: Insights from ERPs. Journal of Psychophysiology 2009, 23(2):63-76.
46. Perry A., Bentin S. Mirror activity in the human brain while observing hand movements: A comparison between EEG desynchronization in the mu-range and previous fMRI results. Brain Research 2009, 1282:126-132.
47. Pineda J.A., Hecht E. Mirroring and mu rhythm involvement in social cognition: Are there dissociable subcomponents of theory of mind?. Biological Psychology 2009, 80(3):306-314.
48. Rizzolatti G., Arbib M.A. Language within our grasp. Trends in Neuroscience 1998, 21:188-194.
49. Rizzolatti G., Craighero L. The mirror-neuron system. Neuroscience 2004, 27:169-192.
50. Rizzolatti G., Fadiga L., Gallese V., Fogassi L. Premotor cortex and the recognition of motor actions. Cognitive Brain Research 1996, 3:131-141.
51. Rogers W.T. The contribution of kinesic illustrators toward the comprehension of verbal behavior within utterances. Human Communication Research 1978, 5:54-62.
52. Schendan H.E., Kutas M. Neurophysiological evidence for two processing times for visual object identification. Neuropsychologia 2002, 40:931-945.
53. Schendan H.E., Kutas M. Time course of processes and representations supporting visual object identification and memory. Journal of Cognitive Neuroscience 2003, 15(1):111-135.
54. Schendan H.E., Kutas M. Neurophysiological evidence for the time course of activation of global shape, part, and local contour representations during visual object categorization and memory. Journal of Cognitive Neuroscience 2007, 19(5):734-749.
55. Schendan H.E., Maher S.M. Object knowledge during entry-level categorization is activated and modified by implicit memory after 200ms. NeuroImage 2009, 44:1423-1438.
56. Sitnikova T., Kuperberg G., Holcomb P.J. Semantic integration in videos of real-world events: An electrophysiological investigation. Psychophysiology 2003, 40(1):160-164.
57. Urbach T.P., Kutas M. The intractability of scaling scalp distributions to infer neuroelectric sources. Psychophysiology 2002, 39:791-808.
58. van Berkum J.J., Hagoort P., Brown C.M. Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience 1999, 11(6):657-671.
59. West W.C., Holcomb P.J. Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research 2002, 13(3):363-375.
60. Willems R.M., Ozyurek A., Hagoort P. When language meets action: The neural integration of gesture and speech. Cerebral Cortex 2007, 17:2322-2333.
61. Willems R.M., Ozyurek A., Hagoort P. Seeing and hearing meaning: ERP and fMRI evidence of word versus picture integration into a sentence context. Journal of Cognitive Neuroscience 2008, 20(7):1235-1249.
62. Willems R.M., Ozyurek A., Hagoort P. Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. NeuroImage 2009, 47:1992-2004.
63. Wu, Y. C., & Coulson, S. (2010b). Iconic gestures facilitate word and message processing: The Multi-Level Integration model of audio-visual discourse comprehension. Poster presented at the Neurobiology of Language Conference, Rancho Bernardo, California.
64. Wu Y.C., Coulson S. Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 2005, 42(6):654-667.
65. Wu Y.C., Coulson S. How iconic gestures enhance communication: An ERP study. Brain and Language 2007, 101(3):234-245.
66. Wu Y.C., Coulson S. Iconic gestures prime related concepts: An ERP study. Psychonomic Bulletin & Review 2007, 14(1):57-63.
67. Wu Y.C., Coulson S. Gestures modulate speech processing early in utterances. NeuroReport 2010, 21(7):522-526.