Learning to attend: A connectionist model of situated language comprehension

Cognitive Science

Volumn 33, Issue 3, 2009, Pages 449-496

  Mayberry, Marshall R ^a   Crocker, Matthew W ^a   Knoeferle, Pia ^b  

^a SAARLAND UNIVERSITY   (Germany)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Attention; Connectionist modeling; Eye tracking; Situated language comprehension; Visual world

Indexed keywords

EID: 72249085412     PISSN: 03640213     EISSN: None     Source Type: Journal    
DOI: 10.1111/j.1551-6709.2009.01019.x     Document Type: Article

References (55)

1. Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73 (3), 247-264.
2. Bailey, D., Feldman, J. A., Narayanan, S., & Lakoff, G. (1997). Modeling embodied lexical development. In M. G. Shafto & P. Langley (Eds.), Proceedings of the 19th annual conference of the Cognitive Science Society (pp. 19-24). Hillsdale, NJ: Lawrence Erlbaum.
3. Barsalou, L. W. (1999). Language comprehension: Archival memory or preparation for situated action? Discourse Processes, 28, 61-80.
4. Cave, K. R. (1999). The FeatureGate model of visual selection. Psychological Research, 62 (2), 182-194.
5. Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687-696.
6. Chang, F., Dell, G. S., & Bock, K. (2006). Becoming syntactic. Psychological Review, 113 (2), 234-272.
7. Christiansen, M. H., & Dale, R. A. C. (2001). Integrating distributional, prosodic and phonological information in a connectionist model of language acquisition. In Proceedings of the 23rd annual conference of the Cognitive Science Society (pp. 220-225). Mahawah, NJ: Lawrence Erlbaum.
8. Christiansen, M. H., Dale, R. A. C., & Reali, F. (in press). Connectionist explorations of multiple-cue integration in syntax acquisition. In S. P. Johnson (Ed.), Neoconstructivism: The new science of cognitive development. New York: Oxford University Press.
9. Cooper, R. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107.
10. Crocker, M. W. (1996). Computational psycholinguistics: An interdisciplinary approach to the study of language. Dordrecht: Kluwer.
11. Crocker, M. W. (2005). Rational models of comprehension: Addressing the performance paradox. In A. Cutler (Ed.), Twenty-first century psycholinguistics: four cornerstones (pp. 363-380). Hillsdale, NJ: Lawrence Erlbaum.
12. Crocker, M. W., & Brants, T. (2000). Wide-coverage probabilistic sentence processing. Journal of Psycholinguistic Research, 29 (6), 647-669.
13. Dominey, P. F., & Boucher, J.-D. (2005). Developmental stages of perception and language acquisition in a perceptually grounded robot. Cognitive Systems Research, 6 (3), 243-259.
14. Dunham, P. J., Dunham, F., & Curwin, A. (1993). Joint-attentional states and lexical acquisition at 18 months. Developmental Psychology, 29, 827-831.
15. Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14 (2), 179-211.
16. Gärdenfors, P. (2000). Conceptual spaces: The geometry of thought. Cambridge, MA: MIT Press.
17. Gibson, J. (1979). The ecological approach to visual perception. Boston: Houghton-Mifflin Co.
18. Gillette, J., Gleitman, H., Gleitman, L., & Lederer, A. (1999). Human simulations of vocabulary learning. Cognition, 73, 135-176.
19. Gray, C. M. (1999). The temporal correlation hypothesis of visual feature integration: Still alive and well. Neuron, 24 (1), 31-47.
20. Hale, J. (2003). The information conveyed by words in sentences. Journal of Psycholinguistic Research, 32 (1), 101-122.
21. Harris, M., Jones, D., Brookes, S., & Grant, J. (1986). Relations between the non-verbal context of maternal speech and rate of language development. British Journal of Developmental Psychology, 4, 261-268.
22. SAIM: A model of visual attention and neglect
23. Jurafsky, D. (1996). A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science, 20, 137-194.
24. Kamide, Y., Altmann, G. T., & Haywood, S. (2003a). Prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language, 49, 133-156.
25. Kamide, Y., Scheepers, C., & Altmann, G. T. (2003b). Integration of syntactic and semantic information in predictive processing: Cross-linguistic evidence from German and English. Journal of Psycholinguistic Research, 32 (1), 37-55.
26. Knoeferle, P., & Crocker, M. W. (2006). The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye-tracking. Cognitive Science, 30 (3), 481-529.
27. Knoeferle, P., & Crocker, M. W. (2007). The inuence of recent scene events on spoken comprehension: Evidence from eye movements. Journal of Memory and Language (Special Issue on Language-Vision Interaction), 57 (4), 519-543.
28. Knoeferle, P., Crocker, M. W., Scheepers, C., & Pickering, M. J. (2005). The inuence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95, 95-127.
29. Knoeferle, P., Habets, B., Crocker, M. W., & Münte, T. F. (2008). Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18 (4), 789-795.
30. Landauer, T. K. (1998). Learning and representing verbal meaning: Latent semantic analysis theory. Current Directions in Psychological Science, 7, 161-164.
31. Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106 (3), 1126-1177.
32. Mayberry, M. R., Crocker, M. W., & Knoeferle, P. (2005). A connectionist model of sentence comprehension in visual worlds. In B. G. Bara, L. Barsalou & M. Bucciarelli (Eds.), Proceedings of the 27th annual conference of the Cognitive Science Society (pp. 1437-1442). Hillsdale, NJ: Lawrence Erlbaum.
33. Mayberry, M. R., Crocker, M. W., & Knoeferle, P. (2006). A connectionist model of the coordinated interplay of scene, utterance, and world knowledge. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th annual conference of the Cognitive Science Society (pp. 567-572). Hillsdale, NJ: Lawrence Erlbaum.
34. McRae, K., Spivey-Knowlton, M. J., & Tanenhaus, M. K. (1998). Modeling the inuence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38 (3), 283-312.
35. Reali, F., Christiansen, M. H., & Monaghan, P. (2003). Phonological and distributional cues in syntax acquisition: scaling up the connectionist approach to multiple-cue integration. In R. Alterman & David Kirsh (Eds.), Proceedings of the 25th annual conference of the Cognitive Science Society (pp. 970-975). Mahawah, NJ: Lawrence Erlbaum.
36. Regier, T. (1996). The human semantic potential: Spatial language and constrained connectionism. Cambridge, MA: MIT Press.
37. Reynolds, J. H., & Desimone, R. (1999). The role of neural mechanisms of attention in solving the binding problem. Neuron, 24 (1), 19-29.
38. Rosch, E. (1978). Principles of categorization. In E. Rosch & B. B. Lloyd (Eds.), Cognition and categorization (pp. 27-48). Hillsdale, NJ: Lawrence Erlbaum.
39. Roskies, A. L. (1999). The binding problem. Neuron, 24 (1), 7-9.
40. Roy, D., & Mukherjee, N. (2005). Towards situated speech understanding: Visual context priming of language models. Computer Speech and Language, 19 (2), 227-248.
41. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart & J. L. McClelland (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition, Volume 1: foundations (pp. 318-362). Cambridge, MA: MIT Press.
42. Sedivy, J. C., Tanenhaus, M. K., Chambers, C. G., & Carlson, G. N. (1999). Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-148.
43. Singer, W. (1999). Neuronal synchrony: A versatile code for the definition of relations? Neuron, 24 (1), 49-65.
44. Siskind, J. M. (2001). Grounding the lexical semantics of verbs in visual perception using force dynamics and event logic. Journal of AI Research, 15, 31-90.
45. Snedeker, J., & Trueswell, J. C. (2004). The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adult sentence processing. Cognitive Psychology, 49, 238-299.
46. Snow, C. E. (1977). Mothers' speech research: From input to interaction. In C. Snow & C. Ferguson (Eds.), Talking to children: Language input and acquisition (pp. 31-49). Cambridge, MA: Cambridge University Press.
47. Spivey, M. J., Richardson, D., & Fitneva, S. (2004). Thinking outside the brain: Spatial indices to linguistic and visual information. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 161-189). New York: Psychology Press.
48. Spivey, M. J., Tanenhaus, M. K., Eberhard, K. M., & Sedivy, J. C. (2002). Eye-movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481.
49. St. John, M. F. (1992). Learning language in the service of a task. In M. Ringle (Ed.), Proceedings of the 14th annual conference of the Cognitive Science Society (pp. 271-276). Hillsdale, NJ: Lawrence Erlbaum.
50. Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.
51. Taylor, J., Hartley, M., & Taylor, N. (2005). Attention as sigma-pi controlled ACh-based feedback. In Proceedings of the international joint conference of neural networks (pp. 256-261). Oxford, UK: Elsevier Science Ltd.
52. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136.
53. Trueswell, J. C., Sekerina, I., Hill, N., & Logrip, M. (1999). The kindergarten-path effect: Studying online sentence processing in young children. Cognition, 73, 89-134.
54. Wolfe, J. M., & Cave, K. R. (1999). The psychophysical evidence for a binding problem in human vision. Neuron, 24 (1), 11-17.
55. Yu, C., Ballard, D. H., & Aslin, R. N. (2005). The role of embodied intention in early lexical acquisition. Cognitive Science, 29, 961-1005.