On the mental representations originating during the interaction between language and vision

Cognitive Processing

Volumn 11, Issue 4, 2010, Pages 295-305

  Mishra, Ramesh Kumar ^c   Marmolejo Ramos, Fernando ^a,b  

^a UNIVERSITY OF ADELAIDE   (Australia)

^b UNIVERSIDAD DEL VALLE   (Colombia)

^c NONE

Author keywords

Embodied cognition; Eye movements; Language comprehension; Visual attention

Indexed keywords

ASSOCIATION; ATTENTION; COGNITION; COMPREHENSION; EYE MOVEMENT; LANGUAGE PROCESSING; LINGUISTICS; MOTOR PERFORMANCE; PRIORITY JOURNAL; RESPONSE TIME; REVIEW; SENSORIMOTOR CORTEX; SENSORIMOTOR FUNCTION; VISION; VISUAL INFORMATION; VISUAL SYSTEM;

ATTENTION; COGNITION; EYE MOVEMENTS; HUMANS; LANGUAGE; VISION, OCULAR; VISUAL PERCEPTION;

EID: 78649694073     PISSN: 16124782     EISSN: 16124790     Source Type: Journal    
DOI: 10.1007/s10339-010-0363-y     Document Type: Review

References (100)

1. K Alaerts SP Swinnen N Wenderoth 2009 Interaction of sound and sight during action perception: evidence for shared modality-dependent action representations Neuropsychologia 47 2593 2599 19463837
2. P Allopenna J Magnuson M Tanenhaus 1998 Tracking the time course of spoken-word recognition using eye movements: evidence for continuous mapping models J Memory Lang 38 419 439
3. GTM Altmann Y Kamide 1999 Incremental interpretation at verbs: restricting the domain of subsequent reference Cognition 73 247 264 1:STN:280:DC%2BD3c%2FlsVajtg%3D%3D 10585516
4. GTM Altmann Y Kamide 2007 The real-time mediation of visual attention by language and world knowledge: linking anticipatory (and other) eye movements to linguistic processing J Memory Lang 57 502 518
5. GTM Altmann 2004 Language-mediated eye movements in the absence of a visual world: the 'blank screen paradigm' Cognition 93 79 87
6. Altmann GTM, Kamide Y (2004) Now you see it, now you don't: mediating the mapping between language and the visual world. In: Henderson J, Ferreira F (eds) The integration of language, vision and action. Psychology Press, Newyork, pp 347-386
7. GTM Altmann Y Kamide 2009 Discourse-mediation of the mapping between language and the visual world Cognition 111 1 55 71 19193366
8. GTM Altmann J Mirkovic 2009 Incrementality and prediction in human sentence processing Cogn Sci 33 4 583 609 20396405
9. SE Anderson MJ Spivey 2009 The enactment of language: decades of interactions between linguistic and motor processes Lang Cogn 1 1 87 111
10. M Arai R Van Gompel C Scheepers 2007 Priming distransitive structures in comprehension Cogn Psychol 54 3 218 250 16973144
11. L Barsalou 2008 Grounded cognition Annu Rev Psychol 59 17 45
12. LW Barsalou 1999 Perceptual symbol systems Behav Brain Sci 22 577 660 1:STN:280:DC%2BD3M3ivFGguw%3D%3D 11301525
13. LW Barsalou WK Simmons AK Barbey CD Wilson 2003 Grounding conceptual knowledge in modality-specific systems Trends Cogn Sci 7 2 84 91 12584027
14. I Biederman 1972 Perceiving real-world scenes Science 177 77 80 1:STN:280:DyaE383ivFOgtA%3D%3D 5041781
15. Biederman I (1981) On the semantics of a glance at a scene. In: Kubovy M, Pomerantz JR (eds) Perceptual organization. Lawrence Erlbaum, Hillsdale, New Jersey, pp 213-263
16. C Bonfiglioli C Finocchiaro B Gesierich F Rositani M Vescovi 2009 A kinematic approach to the conceptual representations of this and that Cognition 111 2 270 274 19285660
17. AM Borghi AM Glenberg MP Kaschak 2004 Putting words in perspective Mem Cogn 32 6 863 873
18. Buswell GT (1935) How people look at pictures. University of Chicago Press, Chicago
19. A Casile MA Giese 2006 Nonvisual motor training influences biological motion perception Curr Biol 16 69 74 1:CAS:528:DC%2BD28XksVahtw%3D%3D 16401424
20. MS Castelhano ML Mack JM Henderson 2009 Viewing task influences eye movement control during active scene perception J Vision 9 3 1 15
21. U Castiello 1999 Mechanisms of selection for the control of hand action Trends Cogn Sci 3 7 264 271 10377541
22. CG Chambers MK Tanenhaus JS Magnuson 2004 Actions and affordances in syntactic ambiguity resolution J Exp Psychol Learn Mem Cogn 30 3 687 696 15099136
23. CG Chambers MK Tanenhaus KM Eberhard H Filip GN Carlson 2002 Circumscribing referential domains during real time language comprehension J Mem Lang 47 1 30 49
24. RM Cooper 1974 The control of eye fixation by the meaning of spoken language: a new methodology for the real-time investigation of speech perception, memory, and language processing Cogn Psychol 6 1 84 107
25. MW Crocker P Knoeferle MR Mayberry 2010 Situated sentence processing: the coordinated interplay account and a neurobehavioral model Brain Lang 112 3 189 201 19450874
26. D Dahan D Swingley M Tanenhaus JS Magnuson 2000 Linguistic gender and spoken word recognition in French J Mem Lang 42 465 480
27. De Graef P (2005) Gaze control in scenes. In: Underwood G (ed) Cognitive processes in eye guidance. Oxford University Press, Oxford, pp 189-211
28. De Graef P, Lauwereyns J, Verfaillie K (2000) Attentional orienting and scene semantics (Psyc.Rep.No.268). University of Leuven, Laboratory of Experimental Psychology, Leuven
29. KM Eberhard MJ Spivey-Knowlton JC Sedivy MK Tanenhaus 1995 Eye movements as a window into real-time language comprehension in natural context J Psycholinguist Res 24 6 409 436 1:STN:280:DyaK287gt1yktQ%3D%3D 8531168
30. TA Farmer SE Anderson M Spivey 2007 Gradiency and visual context in syntactic garden-paths J Mem Lang 57 4 570 595 18037980
31. L Fei-Fei A Iyer C Koch P Perona 2007 What do we perceive in a glance of a real world scene? J Vision 7 1 1 29
32. F Ferreira JM Henderson 2007 Introduction to the special issue of language-vision interaction J Mem Lang 57 455 459
33. Findlay JM, Gilchrist ID (2001) Visual attention: the active vision perspective. In: Jenkins M, Harris L (eds) Vision and attention. Springer, New York, pp 83-103
34. M Finkbeiner JH Song K Nakayama A Caramazza 2008 Engaging the motor system with masked orthographic primes: a kinematic analysis Vis Cogn 16 1 11 22
35. Gibson JJ (1977) The theory of affordances. In: Shaw RE, Bransford J (eds) Perceiving, acting, and knowing. Towards an ecological psychology. Lawrence Erlbaum Associates, Hillsdale, NJ, pp 67-82
36. Gibson JJ (1979) The ecological approach to visual perception. Lawrence Erlbaum, Hillsdale, NJ
37. Givón T (2002) The visual information-processing system as an evolutionary precursor of human language. In: Givón T, Malle BF (eds) The evolution of language out prelanguage. John Benjamins, Amsterdam, pp 3-50
38. AM Glenberg 1997 What memory is for? Behav Brain Sci 20 1 55 1:STN:280:DyaK1M7psFGgsg%3D%3D 10096994
39. AM Glenberg DA Robertson 1999 Indexical understanding of instructions Discourse Processes 28 1 26
40. AM Glenberg R Becker S Koltzer L Kolanko S Muller M Rinke 2009 Episodic affordances contribute to language comprehension Lang Cogn 1 1 113 135
41. MR Green A Oliva 2009 Recognition of natural scenes from global properties: seeing the forest without representing the trees Cogn Psychol 58 137 176
42. AG Greenwald 1970 Sensory feedback mechanisms in performance control: with special reference to the ideo-motor mechanism Psychol Rev 77 2 73 99 1:STN:280:DyaE3c3nvVCmtQ%3D%3D 5454129
43. O Hauk I Johnsrude F Pulvermüller 2004 Somatotopic representation of action words in human motor and premotor cortex Neuron 41 301 307 1:CAS:528:DC%2BD2cXhtVahsbs%3D 14741110
44. JM Henderson 2003 Human gaze control during real-world scene perception Trends Cogn Sci 7 11 498 504 14585447
45. JM Henderson 2007 Regarding scenes Curr Dir Psychol Sci 16 411 430
46. Henderson JM, Ferreira F (2004) Scene perception for psycholinguists. In: Henderson JM, Ferreira F (eds) The interface of language, vision and action. Psychology Press, Hove, pp 1-58
47. JM Henderson A Hollingworth 1999 High-level scene perception Annu Rev Psychol 50 243 271 1:STN:280:DyaK1M7nsVanug%3D%3D 10074679
48. JM Henderson GL Pierce 2008 Eye movements during scene viewing: evidence for mixed control of fixation durations Psychon Bull Rev 15 566 573 18567256
49. F Huettig GTM Altmann 2007 Visual-shape computation during language mediated attention is based on lexical input and not modulated by contextual appropriateness Vis Cogn 15 8 985 1018
50. F Huettig PT Quinlan SA McDonald GTM Altmann 2006 Models of high dimensional semantic space predicts language-mediated eye movements in the visual world Acta Psychol 121 65 80
51. L Itti C Koch 2001 Computational modelling of visual attention Nat Rev Neurosci 2 3 194 203 1:CAS:528:DC%2BD3MXitVyqsLs%3D 11256080
52. L Itti C Koch E Niebur 1998 A model of saliency-based visual attention for rapid scene analysis IEEE Trans Pattern Anal Mach Intell 2 11 1254 1259
53. R Jackendoff 2007 A parallel architecture perspective on language processing Brain Res 1146 2 22 1:CAS:528:DC%2BD2sXkt1ejtrg%3D 17045978
54. E Kaiser JT Runner RS Sussman MK Tanenhaus 2009 Structural and semantic constraints on the resolution of pronouns and reflexives Cognition 112 1 55 80 19426968
55. Y Kamide 2008 Anticipatory processes in sentence processing Lang Linguist Compass 2 4 647 670
56. Y Kamide GTM Altmann SL Haywood 2003 Prediction and thematic information in incremental sentence processing: evidence from anticipatory eye movements J Mem Lang 49 133 156
57. Z Kapoula G Daunys O Herbez Q Yang 2009 Effect of title on eye-movement exploration of cubist paintings by Fernand Léger Perception 38 4 479 491 19522318
58. MP Kaschak AM Glenberg 2000 Constructing meaning: the role of affordances and grammatical constructions in sentence comprehension J Mem Lang 43 508 529
59. MP Kaschak CJ Madden DJ Therriault RH Yaxley M Aveyard AA Blanchard RA Zwaan 2005 Perception of motion affects language processing Cognition 94 3 B79 B89 15617669
60. P Knoeferle M Crocker 2007 The influence of recent scene events on spoken language comprehension: evidence from eye movements J Mem Lang 57 4 519 543
61. P Knoeferle MW Crocker 2006 The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking Cogn Sci 30 481 529
62. P Knoeferle MW Crocker C Scheepers MJ Pickering 2005 The influence of the immediate visual context on incremental thematic role-assignment: evidence from eye-movements in depicted events Cognition 95 95 127 15629475
63. GR Loftus NH Mackworth 1978 Cognitive determinants of fixation location during picture viewing J Exp Psychol Hum Percept Perform 4 4 565 572 1:STN:280:DyaE1M%2FmvFSmsA%3D%3D 722248
64. NH Mackworth AJ Morandi 1967 The gaze selects information details within pictures Percept Psychophys 2 11 547 551
65. F Marmolejo-Ramos 2007 Nuevos avances en el estudio científico de la comprensión de textos Universitas Psychologica 6 2 331 343
66. F Marmolejo-Ramos MR Elosúa de Juan P Gygax C Madden S Mosquera 2009 Reading between the lines: the activation of embodied background knowledge during text comprehension Pragmat Cogn 17 1 77 107
67. Marmolejo-Ramos F, Ibañez A, Mishra R, Yokoyama S (2010) Towards a neurocognitive architecture of language, vision, and emotion comprehension. Manuscript in preparation
68. MEJ Masson DN Bub CM Warren 2008 Kicking calculators: contribution of embodied representations to sentence comprehension J Mem Lang 59 256 265
69. MR Mayberry MW Crocker P Knoeferle 2009 Learning to attend: a connectionist model of situated language comprehension Cogn Sci 33 449 496
70. RK Mishra 2009 Interaction of language and visual attention: evidence from production and comprehension Prog Brain Res 176 277 292 19733763
71. K O'Regan A Noe 2001 A sensorimotor account of vision and visual consciousness Behav Brain Sci 25 5
72. A Papafragou J Hulbert J Trueswell 2008 Does language guide event perception? Evidence from eye movements Cognition 108 155 184 18395705
73. DJ Parkhurst E Neibur 2003 Scene content selected by active vision Spat Vis 16 2 125 154 12696858
74. Purves D, Brannon EM, Cabeza R, Huettel SA, LaBar KS, Platt ML, Woldorff MG (2008) Principles of cognitive neuroscience. Sinauer, Sunderland, MA
75. K Rayner 1998 Eye movements in reading and information processing: 20 years of research Psychol Bull 124 372 422 1:STN:280:DyaK1M%2FmsF2ktg%3D%3D 9849112
76. K Rayner 2009 Eye movements and attention in reading, scene perception, and visual search Q J Exp Psychol 62 8 1457 1506
77. K Rayner X Li CC Williams KR Cave AD Well 2007 Eye movements during information processing tasks: individual differences and cultural effects Vis Res 47 2714 2726 17614113
78. K Rayner TJ Smith GL Malcolm JM Henderson 2008 Eye movements and visual encoding during scene perception Psychol Sci 20 1 6 10 19037907
79. DC Richardson T Matlock 2007 The integration of figurative language and static depictions: an eye movement study of fictive motion Cognition 102 129 138 16434031
80. D Roy 2005 Grounding words in perception and action: computational insights Trends Cogn Sci 9 8 389 396 16006171 (Pubitemid 41074906)
81. S Schütz-Bosbach W Prinz 2007 Perceptual resonance: action-induced modulation of perception Trends Cogn Sci 11 8 349 355 17629544
82. JE Sedivy MK Tanenhaus CG Chambers GN Carlson 1999 Achieving incremental interpretation through contextual representation: evidence from the processing of adjectives Cognition 71 109 147 1:STN:280:DyaK1Mzntl2gsA%3D%3D 10444906
83. S Shimada 2010 Deactivation in the sensorimotor area during observation of a human agent performing robotic actions Brain Cogn 72 394 399 20031292
84. Singh N, Mishra RK (in press) Simulating motion in figurative language comprehension. Open Neuroimaging J
85. Spivey MJ (2007) The continuity of mind. Oxford University Press, New York
86. M Spivey JJ Geng 2001 Oculomotor mechanisms activated by imagery and memory: eye movements to absent objects Psychol Res 65 4 235 241 1:STN:280:DC%2BD38%2Fmtlyrug%3D%3D 11789427
87. MJ Spivey MK Tanenhaus KM Eberhard JC Sedivy 2002 Eye movements and spoken language comprehension: effects of visual context on syntactic ambiguity resolution Cogn Psychol 45 4 447 481 12480476
88. Talmy L (2000) Towards a cognitive semantics. MIT Press, Cambridge, MA
89. LH Tan AHD Chan P Kay PL Khong LK Yip KK Luke 2008 Language affects patterns of brain activation associated with perceptual decision PNAS 105 10 4004 4009 1:CAS:528:DC%2BD1cXjs1Omtbg%3D 18316728
90. M Tanenhaus M Spivey-Knowlton K Eberhard J Sedivy 1995 Integration of visual and linguistic information in spoken-language comprehension Science 268 1632 1634 1:CAS:528:DyaK2MXmtlarurw%3D 7777863
91. A Torralba A Oliva MS Castelhano JM Henderson 2006 Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search Psychol Rev 113 766 786 17014302
92. Underwood G (ed) (2005) Cognitive guidance in eye movements. Oxford University Press, Oxford
93. G Underwood D Crundall K Hodson 2005 Confirming statements about pictures of natural scenes Perception 32 1069 1084
94. G Vingerhoets K Vandamme A Vercammen 2009 Conceptual and physical object qualities contribute differently to motor affordances Brain Cogn 69 481 489 1:STN:280:DC%2BD1M7nsl2isw%3D%3D 19046798
95. G Westerman D Mareschal MH Johnson S Sirois MW Spratling MSC Thomas 2007 Neuroconstructivism Dev Sci 10 1 75 83
96. G Westermann S Sirios TR Shultz D Mareschal 2006 Modeling developmental cognitive neuroscience Trends Cogn Sci 10 5 227 232 16603407
97. Yarbus AR (1967) Eye movements and vision. Plenum Press, New York
98. E Yee J Sedivy 2006 Eye movements to pictures reveal transient semantic activation during spoken word recognition J Exp Psychol Learn Mem Cogn 32 1 1 14 16478336
99. GJ Zelinsky 2008 A theory of eye movements during target acquisition Psychol Rev 115 787 835 18954205
100. R Zwann LJ Taylor 2006 Seeing, acting and understanding: motor resonance in language comprehension J Exp Psychol Gen 135 1 1 11