On the temporal dynamics of language-mediated vision and vision-mediated language

Acta Psychologica

Volumn 137, Issue 2, 2011, Pages 181-189

  Anderson, Sarah E ^a   Chiu, Eric ^b   Huette, Stephanie ^b   Spivey, Michael J ^b  

^a Department of Obstetrics Gynecology   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Eye tracking; Psycholinguistics; Visual perception

Indexed keywords

ARTICLE; COMPREHENSION; EYE MOVEMENT; HUMAN; LANGUAGE; LINGUISTICS; PHYSIOLOGY; SPEECH PERCEPTION; VISION;

COMPREHENSION; EYE MOVEMENTS; HUMANS; LANGUAGE; PSYCHOLINGUISTICS; SPEECH PERCEPTION; VISUAL PERCEPTION;

EID: 79957960941     PISSN: 00016918     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.actpsy.2010.09.008     Document Type: Article

References (132)

1. Abrams R.A., Balota D.A. Mental chronometry: Beyond reaction time. Psychological Science 1991, 2:153-157.
2. Aks D., Zelinsky G., Sprott J. Memory across eye-movements: 1/f dynamic in visual search. Nonlinear Dynamics, Psychology, and Life Sciences 2002, 6:1-25.
3. Allopenna P.D., Magnuson J.S., Tanenhaus M.K. Tracking the time course of spoken word recognition: Evidence for continuous mapping models. Journal of Memory and Language 1998, 38:419-439.
4. Altmann G., Steedman M. Interaction with context during human sentence processing. Cognition 1988, 30:191-238.
5. Anderson S.E., Huette S., Matlock T., Spivey M.J. On the temporal dynamics of negated perceptual simulations. Meaning, form, and body 2010, 1-20. CSLI Publications, Stanford. F. Parrill, M. Turner, V. Tobin (Eds.).
6. Anderson S.E., Spivey M.J. The enactment of language: Decades of interactions between linguistic and motor processes. Language and Cognition 2009, 1:87-111.
7. Balota D.A., Abrams R.A. Mental chronometry: Beyond onset latencies in the lexical decision task. Journal of Experimental Psychology: Learning, Memory, and Cognition 1995, 21:1289-1302.
8. Barrett H.C., Kurzban R. Modularity in cognition: Framing the debate. Psychological Review 2006, 113:628-647.
9. Barsalou L.W. Grounded cognition. Annual Review of Psychology 2008, 59:617-645.
10. Bechtel W. Modules, brain parts, and evolutionary psychology. Evolutionary psychology: Alternative approaches 2003, 211-227. Kluwer Academic Publishers, Dordrecht, Netherlands. S. Scher, F. Rauscher (Eds.).
11. Bohm D. Wholeness and the implicate order 1980, Routledge & Kegan Paul, London.
12. Calvert G., Bullmore E., Brammer M., Cambell R., Williams S., McGuire P., et al. Activation of auditory cortex during silent lipreading. Science 1997, 276:593-596.
13. Cavanagh P. Pathways in early vision. Computational processes in human vision: An interdisciplinary perspective 1988, 254-289. Ablex Publishing, Norwood, NJ. Z. Pylyshyn (Ed.).
14. Chambers C., Tanenhaus M., Magnuson J. Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition 2004, 30:687-696.
15. Chemero A.P. Radical embodied cognition 2009, MIT Press, Cambridge, MA.
16. Chiu, E. M. & Spivey M. J. (in preparation). Timing of speech and display affects the linguistic mediation of visual search.
17. Cisek P., Kalaska J.F. Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron 2005, 45:801-814.
18. Cooper R.M. The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology 1974, 6:84-107.
19. Dahan D. The time course of interpretation in speech comprehension. Current Directions in Psychological Science 2010, 19:121-126.
20. Dale R., Kehoe C., Spivey M.J. Graded motor responses in the time course of categorizing atypical exemplars. Memory and Cognition 2007, 35:15-28.
21. Desimone R. Visual attention mediated by biased competition in extrastriate visual cortex. Philosophical Transactions of the Royal Society B: Biological Sciences 1998, 353:1245.
22. Desimone R., Duncan J. Neural mechanisms of selective visual attention. Annual Review of Neuroscience 1995, 18:193-222.
23. Dosher B. Retrieval of sentences from memory: A speed accuracy tradeoff study. Cognitive Psychology 1976, 8:291-310.
24. Dosher B., Han S., Lu Z. Parallel processing in visual search asymmetry. Journal of Experimental Psychology: Human Perception and Performance 2004, 30:3-27.
25. Driver J., Spence C. Crossmodal attention. Current Opinion in Neurobiology 1998, 8:245-253.
26. Duncan J., Humphreys G. Visual search and stimulus similarity. Psychological Review 1989, 96:433-458.
27. Eckstein M. The lower visual search efficiency for conjunctions is due to noise and not serial attentional processing. Psychological Science 1998, 9:111-118.
28. Egeth H., Virzi R., Garbart H. Searching for conjunctively defined targets. Journal of Experimental Psychology: Human Perception and Performance 1984, 10:32-39.
29. Emberson L., Weiss R., Barbosa A., Vatikiotis-Bateson E., Spivey M. Crossed hands curve saccades: Multisensory dynamics in saccade trajectories. Proceedings of the 29th Annual Conference of the Cognitive Science Society 2008.
30. Farah M. Neuropsychological inference with an interactive brain: A critique of the "locality" assumption. Behavioral and Brain Sciences 1994, 17:43-104.
31. Farmer T.A., Anderson S.E., Spivey M.J. Gradiency and visual context in syntactic garden-paths. Journal of Memory & Language 2007, 57:570-595.
32. Ferreira F., Bailey K., Ferraro V. Good-enough representations in language comprehension. Current Directions in Psychological Science 2002, 11:11-15.
33. Finkbeiner M., Caramazza A. Modulating the masked congruence priming effect with the hands and the mouth. Journal of Experimental Psychology: Human Perception and Performance 2008, 34:894-918.
34. Fodor J. The modularity of mind: An essay on faculty psychology 1983, MIT Press, Cambridge, MA.
35. Fraizer L., Rayner K. Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology 1982, 14:178-210.
36. Freeman J.B., Ambady N., Rule N.O., Johnson K.L. Will a category cue attract you? Motor output reveals dynamic competition across person construal. Journal of Experimental Psychology: General 2008, 137:673-690.
37. Gallant J., Connor C., Van Essen D. Neural activity in areas V1, V2 and V4 during free viewing of natural scenes compared to controlled viewing. Neuroreport 1998, 9:85-89.
38. Gallese V., Lakoff G. The brain's concepts: The role of the sensory-motor system in reason and language. Cognitive Neuropsychology 2005, 22:455-479.
39. Gibson B.S., Eberhard K.M., Bryant T.A. Linguistically mediated visual search: The critical role of speech rate. Psychonomic Bulletin & Review 2005, 12:276.
40. Gibson J. The ecological approach to visual perception 1979, Houghton Mifflin, Boston.
41. Giray M., Ulrich R. Motor coactivation revealed by response force in divided and focused attention. Journal of Experimental Psychology: Human Perception and Performance 1993, 19:1278-1291.
42. Gleitman L.R., January D., Nappa R., Trueswell J.C. On the give and take between event apprehension and utterance formulation. Journal of Memory and Language 2007, 57:544-569.
43. Glenberg A.M. Language and action: Creating sensible combinations of ideas. The Oxford handbook of psycholinguistics 2007, 361-370. Oxford University Press, Oxford, UK. G. Gaskell (Ed.).
44. Gold J.I., Shadlen M.N. Representation of a perceptual decision in developing oculomotor commands. Nature 2000, 404:390-394.
45. Goldstone R.L., Lippa Y., Shiffrin R.M. Altering object representations through category learning. Cognition 2001, 78:27-43.
46. Griffin Z., Bock K. What the eyes say about speaking. Psychological Science 2000, 11:274-279.
47. Grosof D., Shapley R., Hawken M. Macaque V1 neurons can signal "illusory" contours. Nature 1993, 365:550-552.
48. Henis E.A., Flash T. Mechanisms underlying the generation of averaged modified trajectories. Biological Cybernetics 1995, 72:407-419.
49. Hommel B. Event files: Feature binding in and across perception and action. Trends in Cognitive Sciences 2004, 8:494-500.
50. Howell S.R., Jankowicz D., Becker S. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. Journal of Memory and Language 2005, 53:258-276.
51. Huette S., McMurray B. Continuous dynamics of color categorization. Psychonomic Bulletin & Review 2010, 17:348-354.
52. Huettig F., Altmann G.T.M. Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition 2005, 96:23-32.
53. Huettig F., Altmann G.T.M. Visual-shape competition and the control of eye fixation during the processing of unambiguous and ambiguous words. Visual Cognition 2007, 15:985-1018.
54. Itti L., Koch C. Computational modeling of visual attention. Nature Reviews Neuroscience 2001, 2:194-203.
55. Kaptein N.A., Theeuwes J., van der Heijden A.H.C. Search for a conjunctively defined target can be selectively limited to a color-defined subset of elements. Journal of Experimental Psychology: Human Perception & Performance 1995, 21:1053-1069.
56. Kello C.T., Beltz B.C., Holden J.G., Van Orden G.C. The emergent coordination of cognitive function. Journal of Experimental Psychology: General 2007, 136:551-568.
57. Kveraga K., Ghuman A.S., Bar M. Top-down predictions in the cognitive brain. Brain and Cognition 2007, 65:145-168.
58. Lalanne C., Lorenceau J. Crossmodal integration for perception and action. Journal of Physiology - Paris 2004, 98:265-279.
59. Lee T.S., Nguyen M. Dynamics of subjective contour formation in the early visual cortex. Proceedings of the National Academy of Science 2001, 98:1907-1911.
60. Liberman A., Harris K., Kinney J., Lane H. The discrimination of relative onset-time of the components of certain speech and nonspeech patterns. Journal of Experimental Psychology 1961, 61:379-388.
61. Lupyan G. The conceptual grouping effect: Categories matter (and named categories matter more). Cognition 2008, 108:566-577.
62. Lupyan G., Spivey M.J. Perceptual processing is facilitated by ascribing meaning to novel stimuli. Current Biology 2008, 18:410-412.
63. Lupyan G., Spivey M.J. Making the invisible visible: Verbal but not visual cues enhance visual detection. PLoS ONE 2010, 5(7):e11452.
64. Macaluso E., Frith C., Driver J. Modulation of human visual cortex by crossmodal spatial attention. Science 2000, 289:1206-1208.
65. Macknik S.L., Fisher B.D., Bridgeman B. Flicker distorts visual space constancy. Vision Research 1991, 31:2057-2064.
66. Magnuson J.S. Moving hand reveals dynamics of thought. Proceedings of the National Academy of Sciences 2005, 102:9995-9996.
67. Mahon B.Z., Caramazza A. A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology - Paris 2008, 102:59-70.
68. Marslen-Wilson M. Functional parallelism in spoken word recognition. Cognition 1987, 25:71-102.
69. Massaro D. Speechreading: Illusion or window into pattern recognition. Trends in Cognitive Sciences 1999, 3:310-317.
70. McClelland J., Elman J. The TRACE model of speech perception. Cognitive Psychology 1986, 18:1-86.
71. McElree B., Carrasco M. The temporal dynamics of visual search: Evidence for parallel processing in feature and conjunction searches. Journal of Experimental Psychology: Human Perception and Performance 1999, 25:1517-1539.
72. McGurk H., MacDonald J. Hearing lips and seeing voices. Nature 1976, 264:746-748.
73. McKinstry C., Dale R., Spivey M.J. Action dynamics reveal parallel competition in decision making. Psychological Science 2008, 19:22-24.
74. McLeod P., Driver J., Crisp J. Visual search for a conjunction of movement and form is parallel. Nature 1988, 332:154-155.
75. McMurray B., Tanenhaus M., Aslin R. Gradient effects of within-category phonetic variation on lexical access. Cognition 2002, 86:B33-B42.
76. McMurray B., Tanenhaus M., Aslin R., Spivey M. Probabilistic constraint satisfaction at the lexical/phonetic interface: Evidence for gradient effects of within-category VOT on lexical access. Journal of Psycholinguistic Research 2003, 32:77-97.
77. McMurray R., Spivey M. The categorical perception of consonants: The interaction of learning and processing. Proceedings of the Chicago Linguistic Society Panels, 35-2 1999, 205-221.
78. Meteyard L., Zokaei N., Bahrami B., Vigliocco G. Now you see it: Visual motion interferes with lexical decision on motion words. Current Biology 2008, 18:R732-R733.
79. Miller J. Discrete versus continuous stage models of human information processing: In search of partial output. Journal of Experimental Psychology: Human Perception and Performance 1982, 8:273-296.
80. Mounts J., Tomaselli R. Competition for representation is mediated by relative attentional salience. Acta Psychologica 2005, 118:261-275.
81. Mumford D. On the computational architecture of the neocortex II. The role of cortico-cortical loops. Biological Cybernetics 1992, 66:241-251.
82. Nakayama K., Silverman G. Serial and parallel processing of visual feature conjunctions. Nature 1986, 320:264-265.
83. Nazir T.A., Boulenger V., Jeannerod M., Paulignan Y., Roy A., Silber B. Language-induced motor perturbations during the execution of a reaching movement. Journal of Cognitive Neuroscience 2007, 18:1607-1615.
84. Neisser U. Cognition and reality: Principles and implications of cognitive psychology 1976, W. H. Freeman, San Francisco, California.
85. Olds E.S., Cowan W.B., Jolicoeur P. Partial orientation pop-out helps difficult search for orientation. Perception & Psychophysics 2000, 62:1341-1347.
86. Olds E.S., Cowan W.B., Jolicoeur P. The time-course of pop-out search. Vision Research 2000, 40:891-912.
87. Olds E.S., Cowan W.B., Jolicoeur P. Tracking visual search over space and time. Psychonomic Bulletin & Review 2000, 7:292-300.
88. Palmer J., Verghese P., Pavel M. The psychophysics of visual search. Vision Research 2000, 40:1227-1268.
89. Pisoni D., Tash J. Reaction times to comparisons within and across phonetic categories. Perception and Psychophysics 1974, 15:285-290.
90. Pulvermüller F. Brain mechanisms linking language and action. Nature Reviews Neuroscience 2005, 6:576-582.
91. Pulvermüller F., Hauk O., Ilmoniemi R., Nikulin V. Functional links between motor and language systems. European Journal of Neuroscience 2005, 21:793-797.
92. Ratcliff R. Theoretical interpretations of the speed and accuracy of positive and negative responses. Psychological Review 1985, 92:212-225.
93. Rauscher F.B., Krauss R.M., Chen Y. Gesture, speech and lexical access: The role of lexical movements in speech production. Psychological Science 1996, 7:226-231.
94. Reali F., Spivey M.J., Tyler M.J., Terranova J. Inefficient conjunction search made efficient by concurrent spoken delivery of target identity. Perception and Psychophysics 2006, 68:959.
95. Reynolds J., Desimone R. Neural mechanisms of attentional selection. Visual attention and cortical circuits 2001, 121-135. MIT Press, Cambridge, Mass. J. Braun, C. Koch, J. Davis (Eds.).
96. Richardson D., Spivey M., Barsalou L., McRae K. Spatial representations activated during real-time comprehension of verbs. Cognitive Science 2003, 27:767-780.
97. Rolls E., Tovee M. Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex. Journal of Neurophysiology 1995, 73:713-726.
98. Sekuler R., Sekuler A., Lau R. Sound alters visual motion perception. Nature 1997, 385:308.
99. Shams L., Kamitani Y., Shimojo S. What you see is what you hear. Nature 2000, 408:788.
100. Siegler R.S., Lin X. Self-explanations promote children's learning. Metacognition, strategy use, and instruction 2009, 85-113. Guilford Publications, New York. H.S. Waters, W. Schneider (Eds.).
101. Snedeker J., Trueswell J. The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adult sentence processing. Cognitive Psychology 2004, 49:238-299.
102. Crossmodal space and crossmodal attention 2004, Oxford University Press, Oxford. C. Spence, J. Driver (Eds.).
103. Spence C., Nicholls M.E.R., Gillespie N., Driver J. Cross-modal links in exogenous covert spatial orienting between touch, audition, and vision. Perception & Psychophysics 1998, 60:544-557.
104. Spivey M.J. The continuity of mind 2007, Oxford University Press, New York.
105. Spivey M.J., Dale R. On the continuity of mind: Toward a dynamical account of cognition. The psychology of learning & motivation. Volume 45 2004, 87-142. Elsevier, San Diego. B. Ross (Ed.).
106. Spivey M.J., Dale R., Knoblich G., Grosjean M. Do curved reaching movements emerge from competing perceptions?. Journal of Experimental Psychology: Human Perception and Performance 2010, 36:251-254.
107. Spivey M.J., Grosjean M., Knoblich G. Continuous attraction toward phonological competitors. Proceedings of the National Academy of Sciences 2005, 102:10393-10398.
108. Spivey M.J., Tanenhaus M., Eberhard K., Sedivy J. Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology 2002, 45:447-481.
109. Spivey M.J., Tyler M.J., Eberhard K.M., Tanenhaus M.K. Linguistically mediated visual search. Psychological Science 2001, 12:282-286.
110. Spivey-Knowlton, M. (1996). Integration of visual and linguistic information: Human data and model simulations. Doctoral dissertation, University of Rochester, Rochester, NY.
111. Spivey-Knowlton M.J., Tanenhaus M., Eberhard K., Sedivy J. Eye movements accompanying language and action in a visual context: Evidence against modularity. Proceedings of the 17th Annual Conference of the Cognitive Science Society 1995, 25-30. Erlbaum, Hillsdale, NJ.
112. Spivey-Knowlton M., Trueswell J., Tanenhaus M. Context effects in syntactic ambiguity resolution. Canadian Journal of Experimental Psychology 1993, 47:276-309.
113. Stephen D.G., Mirman D. Interactions dominate the dynamics of visual cognition. Cognition 2010, 115:154-165.
114. Strack F., Martin L., Stepper S. Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology 1988, 54:768-777.
115. Tanenhaus M., Spivey-Knowlton M., Eberhard K., Sedivy J. Integration of visual and linguistic information during spoken language comprehension. Science 1995, 268:1632-1634.
116. Theeuwes J., Kooi F. Parallel search for a conjunction of contrast polarity and shape. Vision Research 1994, 34:3013-3016.
117. Theeuwes J., Olivers C.N.L., Chizk C.L. Remembering a location makes the eyes curve away. Psychological Science 2005, 16:196-199.
118. Tipper S., Houghton G., Howard L. Behavioural consequences of selection from neural population codes. Control of cognitive processes: Attention and performance XVIII 2000, 233-245. MIT, Cambridge, MA. S. Monsell, J. Driver (Eds.).
119. Tipper S., Howard L., Jackson S. Selective reaching to grasp: Evidence for distractor interference effects. Visual Cognition 1997, 4:1-38.
120. Todd J.W. Reaction to multiple stimuli. Archives of Psychology 1912, 25:1-65.
121. Tomlin R. Mapping conceptual representations into linguistic representations: The role of attention in grammar. Language and conceptualization 1997, 162-189. Cambridge University Press, Cambridge. J. Nuyts, E. Pederson (Eds.).
122. Treisman A.M., Gelade G. A feature-integration theory of attention. Cognitive Psychology 1980, 12:97-136.
123. Trueswell J., Sekerina K., Hill, Logrip L. The kindergarten-path effect: Studying on-line sentence processing in young children. Cognition 1999, 73:89-134.
124. Tucker M., Ellis R. The potentiation of grasp types during visual object categorization. Visual Cognition 2001, 8:769-800.
125. Turvey M.T. Action and perception at the level of synergies. Human Movement Science 2007, 26:657-697.
126. van der Wel R.P.R.D., Eder J., Mitchel A., Walsh M., Rosenbaum D. Trajectories emerging from discrete versus continuous processing models in phonological competitor tasks. Journal of Experimental Psychology: Human Perception and Performance 2009, 32:588-594.
127. Van Orden G., Holden J., Turvey M. Self-organization of cognitive performance. Journal of Experimental Psychology: General 2003, 132:331-350.
128. Wojnowicz M., Ferguson M., Dale R., Spivey M.J. The self-organization of explicit attitudes. Psychological Science 2009, 20:1428-1435.
129. Wolfe J.M. Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin & Review 1994, 1:202-238.
130. Wolfe J.M. What can 1million trials tell us about visual search?. Psychological Science 1998, 9:33-39.
131. Yee E., Sedivy J. Eye movements reveal transient semantic activation during spoken word recognition. Journal of Experimental Psychology: Learning, Memory and Cognition 2006, 32:1-14.
132. Zwitserlood P. The locus of the effects of sentential-semantic context in spoken-word processing. Cognition 1989, 32:25-64.