Situated sentence processing: The coordinated interplay account and a neurobehavioral model

Brain and Language

Volumn 112, Issue 3, 2010, Pages 189-201

  Crocker, Matthew W ^a   Knoeferle, Pia ^b   Mayberry, Marshall R ^c  

^a SAARLAND UNIVERSITY   (Germany)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Cognitive modeling; Connectionist modeling; Embodiment; Event related potentials; Language processing; Situated cognition; Visual attention

Indexed keywords

ARTICLE; BRAIN CORTEX; BRAIN FUNCTION; COMPREHENSION; COORDINATION; EVENT RELATED POTENTIAL; EYE MOVEMENT; EYE TRACKING; LANGUAGE ABILITY; LANGUAGE PROCESSING; LINGUISTICS; NERVOUS SYSTEM ELECTROPHYSIOLOGY; NEUROPSYCHOLOGY; PSYCHOLOGICAL MODEL; VERBAL COMMUNICATION; VISUAL DISCRIMINATION;

ATTENTION; BRAIN; COMPREHENSION; HUMANS; LINGUISTICS; MODELS, NEUROLOGICAL;

EID: 77649336346     PISSN: 0093934X     EISSN: 10902155     Source Type: Journal    
DOI: 10.1016/j.bandl.2009.03.004     Document Type: Article

References (114)

1. Allopenna P.D., Magnuson J.S., and Tanenhaus M.K. Tracking the time course of spoken word recognition using eye-movements: Evidence for continuous mapping models. Journal of Memory and Language 38 (1998) 419-439
2. Altmann G.T.M. Thematic role assignment in context. Journal of Memory and Language 41 (1999) 124-145
3. Altmann G.T.M. Language-mediated eye-movements in the absence of a visual world: The blank screen paradigm. Cognition 93 (2004) B79-B87
4. Altmann G.T.M., and Kamide Y. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73 3 (1999) 247-264
5. Altmann G.T.M., and Kamide Y. Now you see it, now you don't: Mediating the mapping between language and the visual world. In: Henderson J., and Ferreira F. (Eds). The integration of language vision and action: Eye movements and the visual world (2004), Psychology Press, New York 347-386
6. Arnold J., Eisenband J., Brown-Schmidt S., and Trueswell J. The rapid use of gender information: Evidence of the time course of pronoun resolution from eyetracking. Cognition 76 (2000) B13-B26
7. Ballard D., Hayhoe M., Pook P., and Rao R. Deictic codes for the embodiment of cognition. Behavioral and Brain Sciences 20 (1997) 723-767
8. Barsalou L.W. Language comprehension: Archival memory or preparation for situated ?. Discourse Processes 28 (1999) 61-80
9. Barsalou L.W. Perceptual symbol systems. Behavioral and Brain Sciences 22 (1999) 577-660
10. beim Graben P., Schlesewsky M., Saddy D., and Kurths J. Symbolic dynamics of event-related brain potentials. Physical Review E 62 (2000) 5518-5541
11. Bever T. The cognitive basis for linguistic structures. In: Hayes J.R. (Ed). Cognition and the development of language (1970), Wiley, New York 279-362
12. Bornkessel I., McElree B., Schlesewsky M., and Friederici A. Multi-dimensional contributions to garden path strength: Dissociating phrase structure from case marking. Journal of Memory and Language 51 (2004) 495-522
13. Bornkessel I., and Schlesewsky M. The extended argument dependency model: A neurocognitive approach to sentence comprehension across languages. Psychological Review 113 (2006) 787-821
14. Bornkessel I., Schlesewsky M., and Friederici A. Grammar overrides frequency: Evidence from the online processing of flexible word order. Cognition 85 (2002) B21-B30
15. Bornkessel I., Schlesewsky M., and Friederici A. The neurophysiological basis of word order variations in german. Brain and Language 86 (2003) 116-128
16. Buccino G., Riggio L., Melli G., Binkofski F., Gallese V., and Rizzolattii G. Listening to action-related sentences modulates the activity of the motor system: A combined TMS and behavioral study. Cognitive Brain Research 24 (2005) 355-363
17. Carpenter P.A., Just M., Keller T., Eddy W., and Thulborn K. Time course of fMRI-activation in language and spatial networks during sentence comprehension. Neuroimage 10 (1992) 216-224
18. Chambers C.G., Tanenhaus M.K., and Magnuson J.S. Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition 30 (2004) 687-696
19. Chater N., and Manning C.D. Probabilistic models of language processing and acquisition. Trends in Cognitive Science 10 7 (2006) 335-344
20. Christiansen M.H., and Chater N. Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23 2 (1999) 157-205
21. Claus B., and Kelter S. Comprehending narratives containing flashbacks: Evidence for temporally organized representations. Journal of Experimental Psychology: Learning, Memory, and Cognition 32 (2006) 1031-1044
22. Cooper R. The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology 6 (1974) 84-107
23. Crocker M.W. Computational psycholinguistics: An interdisciplinary approach to the study of language (1996), Kluwer, Dordrecht
24. Crocker M.W. Mechanisms for sentence processing. In: Garrod S., and Pickering M.J. (Eds). Language processing (1999), Psychology Press, London 191-232
25. Crocker M.W. Rational models of comprehension: Addressing the performance paradox. In: Cutler A. (Ed). Twenty-first century psycholinguistics: Four cornerstones (2005), Lawrence Erlbaum, Hillsdale, NJ 363-380
26. Crocker M.W., and Brants T. Wide-coverage probabilistic sentence processing. Journal of Psycholinguistic Research 29 6 (2000) 647-669
27. Dahan D., Magnuson J., and Tanenhaus M. Time course of frequency effects in spoken-word recognition: Evidence from eye movements. Cognitive Psychology 42 (2001) 361-367
28. DeLong K.A., Urbach T.P., and Kutas M. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience 8 89 (2005) 1117-1121
29. Demarais A., and Cohen B.H. Evidence for image-scanning eye movements during transitive inference. Biological Psychology 49 (1998) 229-247
30. Dunham P.J., Dunham F., and Curwin A. Joint-attentional states and lexical acquisition at 18 months. Developmental Psychology 29 (1993) 827-831
31. Elman J.L. Finding structure in time. Cognitive Science 14 2 (1990) 179-211
32. Farmer T.A., Anderson S.E., and Spivey M.J. Gradiency and visual context in syntactic garden-paths. Journal of Memory and Language (Special Issue on Language-Vision Interaction) 57 4 (2007) 570-595
33. Federmeier K. Thinking ahead: The role and roots of prediction in language comprehension. Psychophysiology 44 (2007) 491-505
34. Fischer M.H., and Zwaan R.A. Embodied language: A review of the role of the motor system in language comprehension. Quarterly Journal of Experimental Psychology 61 6 (2008) 825-850
35. Fodor J. The modularity of mind (1983), MIT Press, Cambridge, MA
36. Fodor J., and Frazier L. The sausage machine: A new two-stage parsing model. Cognition 6 (1978) 291-325
37. Frazier, L. (1979). On comprehending sentences: Syntactic parsing strategies. Unpublished doctoral dissertation, University of Connecticut.
38. Friederici A.D. Towards a neural basis of auditory sentence processing. Trends in Cognitive Science 6 2 (2002) 78-84
39. Frisch S., Schlesewsky M., Saddy D., and Alpermann J. The P600 and an indicator of syntactic ambiguity. Cognition 85 (2002) B83-B92
40. Ganis G., Kutas M., and Sereno M.I. The search for common sense: An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience 8 (1996) 89-106
41. Gibson E. Linguistic complexity: Locality of syntactic dependencies. Cognition 68 (1998) 1-76
42. Glenberg A., and Kaschak M.P. Grounding language in action. Psychonomic Bulletin and Review 9 3 (2002) 558-565
43. Glenberg A., and Robertson D.A. Indexical understanding of instructions. Discourse Processes 28 (1999) 1-26
44. Hagoort P. How the brain solves the binding problem for language: A neurocomputational model of syntactic processing. Neuroimage 20 (2003) S18-S29
45. Hale J. The information conveyed by words in sentences. Journal of Psycholinguistic Research 32 1 (2003) 101-122
46. Harris M., Jones D., Brookes S., and Grant J. Relations between the non-verbal context of maternal speech and rate of language development. British Journal of Developmental Psychology 4 (1986) 261-268
47. Haupt F., Schlesewsky M., Roehm D., Friederici A., and Bornkessel-Schlesewsky I. The status of subject-object reanalysis in the language comprehension architecture. Journal of Memory and Language 59 (2008) 54-96
48. Hennon E., Chung H., and Brown E. What does it take for 12-month-olds to learn a word. Breaking the language barrier: An emergentist coalition model of the origins of word learning (monographs of the society for research in child development) (2000), Blackwell Publishing, Oxford 62-84
49. Jackendoff R. Foundations of language: Brain, meaning, grammar, evolution (2002), Oxford University Press, Oxford, UK
50. Johanssson R., Holsanova J., and Holmqvist K. Pictures and spoken descriptions elicit similar eye movements during mental imagery, both in light and in complete darkness. Cognitive Science 30 6 (2006) 1053-1080
51. Jurafsky D. A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science 20 (1996) 137-194
52. Kamide Y., Scheepers C., and Altmann G.T. Integration of syntactic and semantic information in predictive processing: Cross-linguistic evidence from German and English. Journal of Psycholinguistic Research 32 1 (2003) 37-55
53. Kluender R., and Kutas M. Bridging the gap: Evidence from ERPs on the processing of unbounded dependencies. Journal of Cognitive Neuroscience 5 (1993) 196-214
54. Knoeferle P. Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events. In: van Gompel R., Fischer M., Murray W., and Hill R. (Eds). Eye movement research: Insights into mind and brain (2007), Elsevier, Oxford 517-533
55. Knoeferle P., and Crocker M.W. The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye-tracking. Cognitive Science 30 3 (2006) 481-529
56. Knoeferle P., and Crocker M.W. The influence of recent scene events on spoken comprehension: Evidence from eye movements. Journal of Memory and Language 57 4 (2007) 519-543
57. Knoeferle P., Crocker M.W., Scheepers C., and Pickering M.J. The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition 95 (2005) 95-127
58. Knoeferle P., Habets B., Crocker M.W., and Münte T.F. Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex 18 4 (2008) 789-795
59. Knoeferle, P., Kutas, M., & Urbach, T. (2008). ERP correlates of verb-action and sentence-scene role relations incongruence in a picture-sentence verification task. In Annual meeting of the CNS. Cognitive Neuroscience Society.
60. Konieczny L. Locality and parsing complexity. Journal of Psycholinguistic Research 29 6 (2000) 627-645
61. Kutas M., and Hillyard S.A. Reading senseless sentences: Brain potentials reflect semantic incongruity. Science 207 (1980) 203-205
62. Kutas M., and Hillyard S.A. Event-related brain potentials to grammatical errors and semantic anomalies. Memory and Cognition 11 (1983) 539-550
63. Kutas M., and Hillyard S.A. Brain potentials during reading reflect word expectancy and semantic association. Nature 307 (1984) 161-163
64. Kutas M., van Petten C., and Kluender R. In: Traxler M., and Gernsbacher M. (Eds). Handbook of psycholinguistics (2006), Elsevier 659-724
65. Levy R. Expectation-based syntactic comprehension. Cognition 106 3 (2008) 1126-1177
66. Lupyan, G. (2007). Reuniting categories, language, and perception. In Proceedings of the 29th annual conference of the cognitive science society. Hillsdale, NJ: Erlbaum.
67. MacDonald M.C., Pearlmutter N.J., and Seidenberg M.S. Lexical nature of syntactic ambiguity resolution. Psychological Review 101 (1994) 676-703
68. Marslen-Wilson W.D. Sentence perception as an interactive parallel process. Science 189 (1975) 226-228
69. Matin E., Shao K., and Boff K. Saccadic overhead: Information processing time with and without saccades. Perceptual Psychophysics 53 (1993) 372-380
70. Matzke M., Mai H., Nager W., Rüsseler J., and Münte T. The costs of freedom: An ERP-study of non-canonical sentences. Clinical Neuropsychology 113 (2002) 844-852
71. Mayberry, M. R. (2003). Incremental nonmonotonic parsing through semantic self-organization. Unpublished doctoral dissertation, Department of Computer Sciences. Austin, TX: The University of Texas at Austin.
72. Mayberry M.R., Crocker M.W., and Knoeferle P. Learning to attend: A connectionist model of situated language comprehension. Cognitive Science 33 1 (2009) 449-496
73. Mayberry, M. R., & Miikkulainen, R. (submitted for publication). Incremental nonmonotonic parsing through semantic self-organization.
74. McRae K., Spivey-Knowlton M.J., and Tanenhaus M.K. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language 38 3 (1998) 283-312
75. Osterhout L., and Holcomb P.J. Event-related brain potentials elicited by syntactic anomaly. Journal of Memory and Language 31 (1992) 785-806
76. Osterhout L., and Holcomb P.J. Event-related potentials and syntactic anomaly: Evidence of anomaly brain potentials. Psychophysioloy 30 (1993) 170-182
77. Otten M., Nieuwland M.S., and van Berkum J.J.A. Great expectations: Specific lexical anticipation influences the processing of spoken language. BMC Neuroscience 8 8 (2007) 1-9
78. Pado U., Crocker M., and Keller F. A probabilistic model of semantic plausibility in sentence processing. Cognitive Science (2009) 10.1111/j.1551-6709.2009.01033.x
79. Pickering M.J., and Traxler M. Plausibility and the recovery of garden paths: An eye-tracking study. Journal of Experimental Psychology: Learning, Memory and Cognition 24 (1998) 940-961
80. Pritchett B.L. Grammatical competence and parsing performance (1992), University of Chicago Press, Chicago
81. Pruden S.M., Hirsh-Pasek K., Golinkoff R.M., and Hennon E.A. The birth of words: Ten-month olds learn words through perceptual salience. Psychological Science 18 (2007) 414-420
82. Pulvermüller F. Brain mechanisms linking language and action. Nature Reviews Neuroscience 6 (2005) 576-582
83. Pulvermüller F., Härle M., and Hummel F. Walking or talking?: Behavioural and neurophysiological correlates of action verb processing. Brain and Language 78 (2001) 143-168
84. Richardson D.C., and Spivey M.J. Representation, space and hollywood squares: Looking at things that aren't there anymore. Cognition 76 (2000) 269-295
85. Rösler F., Pechmann T., Streb J., Röder B., and Hennighausen E. Parsing of sentences in a language with varying word order: Word-by-word variations of processing demands are revealed by event-related brain potentials. Journal of Memory and Language 38 (1998) 150-176
86. Rumelhart D.E., Hinton G.E., and Williams R.J. Learning internal representations by error propagation. In: Rumelhart D.E., and McClelland J.L. (Eds). Parallel distributed processing: Explorations in the microstructure of cognition Volume 1: Foundations (1986), MIT Press, Cambridge, MA 318-362
87. Sedivy J.C., Tanenhaus M.K., Chambers C.G., and Carlson G.N. Achieving incremental semantic interpretation through contextual representation. Cognition 71 (1999) 109-148
88. Smith L.B. How to learn words: An associative crane. In: Golinkoff R., and Hirsh-Pasek K. (Eds). Breaking the word learning barrier (2000), Oxford University Press, Oxford 51-80
89. Spivey M.J. The continuity of mind (2007), Oxford University Press, New York
90. Spivey M.J., and Geng J.J. Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research 65 (2001) 235-241
91. Spivey M.J., Richardson D., and Fitneva S. Thinking outside the brain: Spatial indices to linguistic and visual information. In: Henderson J., and Ferreira F. (Eds). The integration of language vision and action (2004), Psychology Press, New York 161-189
92. Spivey M.J., and Richardson D.C. Language embedded in the environment. In: Robbins P., and Aydede M. (Eds). The Cambridge handbook of situated cognition (2009), Cambridge University Press, Cambridge, UK 382-400
93. Spivey M.J., Tyler M.J., Eberhard K.M., and Tanenhaus M.K. Linguistically mediated visual search. Psychological Science 12 4 (2001) 282-286
94. Stanfield R.A., and Zwaan R.A. The effect of impied orientation derived from verbal context on picture recognition. Psychological Science 12 (2001) 153-156
95. Tanenhaus M.K., Spivey-Knowlton M.J., Eberhard K.M., and Sedivy J.C. Integration of visual and linguistic information in spoken language comprehension. Science 268 (1995) 1632-1634
96. Tanenhaus M.K., Spivey-Knowlton M.J., and Hanna J.E. Modelling discourse context effects: A multiple constraints approach. In: Crocker M., Pickering M., and Clifton C. (Eds). Architectures and mechanisms for language processing (2000), Cambridge University Press, Cambridge 90-118
97. Tettamanti M., Buccino G., Saccuman M.C., Gallese V., Danna M., Scifo P., et al. Listening to action-related sentences activates fronto-parietal motor circuits. Journal of Cognitive Neuroscience 17 (2005) 273-281
98. Trueswell J.C., and Tanenhaus M.K. Toward a lexicalist framework for constraint-based syntactic ambiguity resolution. In: Clifton C., Frazier L., and Rayner K. (Eds). Perspectives on sentence processing (1994), Lawrence Erlbaum Associates, Hillsdale, NJ 155-179
99. Trueswell J.C., Tanenhaus M.K., and Garnsey S.M. Semantic influences on parsing: Use of thematic role information in syntactic disambiguation. Journal of Memory and Language 33 (1994) 285-318
100. van Berkum J.J., Brown C.M., Zwitserlood P., Kooijman V., and Hagoort P. Anticipating upcoming words in discourse: Evidence from ERPs and reading times. Journal of Experimental Psychology: Learning, Memory, and Cognition 31 3 (2005) 443-467
101. van Petten C., Coulson S., Rubin S., Plante S., and Parks M. Time course of word identification and semantic integration in spoken language. Journal of Experimental Psychology: Learning, Memory, and Cognition 25 (1999) 394-417
102. van Petten C., and Kutas M. Interactions between sentence context and word frequency in event-related brain potentials. Memory and Cognition 18 (1990) 380-393
103. Vissers C., Kolk H., van de Meerendonk N., and Chwilla D. Monitoring in language perception: Evidence from ERPs in a picture sentence matching task. Neuropsychologia 46 (2008) 967-982
104. Vosse T., and Kempen G. Syntactic structure assembly in human parsing: A computational model based on competitive inhibition and a lexicalist grammar. Cognition 75 (2000) 105-143
105. Wassenaar M., and Hagoort P. Thematic role assignment in patients with Broca's aphasia: Sentence-picture matching electrified. Neuropsychologia 45 (2007) 716-740
106. Weber A., Braun B., and Crocker M.W. Finding referents in time: Eye-tracking evidence for the role of contrastive accents. Language and Speech 49 (2006) 367-392
107. Weber, A., Crocker, & M. W. Knoeferle, P. (in press). Conflicting constraints in resource adaptive language comprehension. In M. W. Crocker, J. Siekmann (Eds.), Resource adaptive cognitive processes. Heidelberg: Springer Verlag.
108. Weber A., Grice M., and Crocker M.W. The role of prosody in the interpretation of structural ambiguities: A study of anticipatory eye movements. Cognition 99 (2006) B63-B72
109. Yaxley R.H., and Zwaan R.A. Simulating visibility during language comprehension. Cognition 105 (2005) B229-B236
110. Zwaan R.A. Embodied cognition, perceptual symbols, and situation models. Discourse Processes 28 1 (1999) 81-88
111. Zwaan R.A. The immersed experiencer: Towards an embodied theory of language comprehension. In: Ross B.H. (Ed). The psychology of learning and motivation Vol. 44 (2004), Academic Press, New York 35-62
112. Zwaan R.A., and Madden C.J. Embodied sentence comprehension in memory, language, and thinking. In: Pecher D., and Zwaan R.A. (Eds). Grounding cognition: The role of perception and action (2005), Cambridge University, Cambridge, UK 224-245
113. Zwaan R.A., Stanfield R.A., and Yaxley R.H. Language comprehenders mentally represent the shapes of objects. Psychological Science 13 2 (2002) 168-171
114. Zwaan R.A., and Taylor L.J. Seeing, acting, understanding: Motor resonance in language comprehension. Journal of Experimental Psychology: General 135 (2006) 1-11