Template construction grammar: From visual scene description to language comprehension and agrammatism

Neuroinformatics

Volumn 12, Issue 1, 2014, Pages 181-208

  Barrès, Victor ^a   Lee, Jinyong ^b  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b University of Southern California   (United States)

Author keywords

Agrammatism; Computational model; Construction grammar; Language comprehension; Language production; Neurolinguistics; Schema theory; Visual scene description

Indexed keywords

ANIMAL; ARTICLE; BIOLOGICAL MODEL; BRAIN; COMPREHENSION; COMPUTER SIMULATION; FACTUAL DATABASE; HUMAN; LANGUAGE; MIRROR NEURON; PHYSIOLOGY; PRIMATE;

ANIMALS; BRAIN; COMPREHENSION; COMPUTER SIMULATION; DATABASES, FACTUAL; HUMANS; LANGUAGE; MIRROR NEURONS; MODELS, NEUROLOGICAL; PRIMATES;

EID: 84894436403     PISSN: 15392791     EISSN: None     Source Type: Journal    
DOI: 10.1007/s12021-013-9197-y     Document Type: Article

References (104)

1. Aine, C. J.; Sanfratello, L.; Ranken, D.; Best, E.; MacArthur, J. A.; et al. (2012). MEG-SIM: a web portal for testing MEG analysis methods using realistic simulated and empirical data. Neuroinformatics, 10, 141-158.
2. Altmann, G. T. M.; & Kamide, Y. (1999). Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition, 73, 247-264.
3. Ansell, B. J.; & Flowers, C. R. (1982). Aphasic adults' use of heuristic and structural linguistic cues for sentence analysis. Brain and Language, 16, 61-72.
4. Arbib, M. A. (1981). Perceptual structures and distributed motor control. In V. B. Brooks (Ed.), Handbook of physiology - The nervous system II. Motor control (pp. 1449-1480). Bethesda: American Physiological Society.
5. Arbib, M. A. (2012). How the brain got language: the mirror system hypothesis. New York & Oxford: Oxford University Press.
6. Arbib, M. A.; & Bota, M. (2003). Language evolution: neural homologies and neuroinformatics. Neural Networks: The Official Journal of the International Neural Network Society, 16, 1237-1260.
7. Arbib, M. A.; & Caplan, D. (1979). Neurolinguistics must be computational. The Behavioral and Brain Sciences, 2, 449-483.
8. Arbib, M. A.; & Lee, J. Y. (2008). Describing visual scenes: towards a neurolinguistics based on construction grammar. Brain Research, 1225, 146-162.
9. Arbib, M. A.; Conklin, E. J.; & Hill, J. C. (1987). From schema theory to language. New York: Oxford University Press. x + 253 pp.
10. Arbib, M. A.; Billard, A.; Iacoboni, M.; & Oztop, E. (2000). Synthetic brain imaging: grasping, mirror neurons and imitation. Neural Networks: The Official Journal of the International Neural Network Society, 13, 975-997.
11. Arbib, M. A.; Plangprasopchok, A.; Bonaiuto, J. J.; Schuler, R. E. (2013). A neuroinformatics of brain modeling and its implementation in the Brain Operation Database BODB. Neuroinformatics, in press.
12. Baker, C.; Fillmore, C. J.; Lowe, J. B. (1998). The {B}erkeley {F}rame{N}et project.
13. Bakker, R.; Wachtler, T.; Diesmann, M. (2012). CoCoMac 2.0 and the future of tract-tracing databases. Frontiers in Neuroinformatics, 6.
14. Barrès, V.; Simons, A.; & Arbib, M. A. (2013). Synthetic event-related potentials: a computational bridge between neurolinguistic models and experiments. Neural Networks, 37, 66-92.
15. Bergen B. K, Chang N. (2005a). Embodied construction grammar in simulation-based language understanding. In Construction Grammars: cognitive grounding and theoretical extensions.
16. Bergen, B. K.; & Chang, N. (2005b). Embodied construction grammar in simulation-based language understanding. In J.-O. OÖstman & M. Fried (Eds.), Construction grammar(s): cognitive and cross-language dimensions (pp. 147-190). Amsterdam: John Benjamins.
17. Berndt, R. S.; & Caramazza, A. (1999). How "regular" is sentence comprehension in Broca's aphasia? It depends on how you select the patients. Brain and Language, 67, 242-247.
18. Berndt, R. S.; Mitchum, C. C.; & Haendiges, A. N. (1996). Comprehension of reversible sentences in "agrammatism": a meta-analysis. Cognition, 58, 289-308.
19. Bornkessel, I.; & Schlesewsky, M. (2006). The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. Psychological Review, 113, 787-821.
20. Caramazza, A.; & Zurif, E. B. (1976). Dissociation of algorithmic and heuristic processes in language comprehension: evidence from aphasia. Brain and Language, 3, 572-582.
21. Caramazza, A.; Capasso, R.; Capitani, E.; & Miceli, G. (2005). Patterns of comprehension performance in agrammatic Broca's aphasia: a test of the trace deletion hypothesis. Brain and Language, 94, 43-53.
22. Chambers, C. G.; Tanenhaus, M. K.; & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology Learning, Memory, and Cognition, 30, 687-696.
23. Chen, R.; Hillis, A. E.; Pawlak, M.; & Herskovits, E. H. (2008). Voxelwise Bayesian lesion-deficit analysis. NeuroImage, 40, 1633-1642.
24. Christianson, K.; & Luke, S. G. (2011). Context strengthens initial misinterpretations of text. Scientific Studies of Reading, 15, 136-166.
25. Christianson, K.; Hollingworth, A.; Halliwell, J. F.; & Ferreira, F. (2001). Thematic roles assigned along the garden path linger. Cognitive Psychology, 42, 368-407.
26. Croft, W. (2001). Radical construction grammar: syntactic theory in typological perspective. Oxford: Oxford University Press.
27. Croft, W.; & Cruse, D. A. (2005). Cognitive linguistics. Cambridge: Cambridge University Press.
28. David, O.; Kiebel, S. J.; Harrison, L. M.; Mattout, J.; Kilner, J. M.; & Friston, K. J. (2006). Dynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30, 1255-1272.
29. De Beule, J.; & Steels, L. (2005). Hierarchy in fluid construction grammar. In U. Furbach (Ed.), Proceedings of the 28th annual German conference on AI, KI 2005, lecture notes in artificial intelligence (vol. 3698) (pp. 1-15). Berlin, Heidelberg: Springer.
30. Dominey, P. F.; & Boucher, J.-D. (2005). Learning to talk about events from narrated video in a construction grammar framework. Artificial Intelligence, 167, 31-61.
31. Dominey, P. F.; Hoen, M.; & Inui, T. (2006a). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088-2107.
32. Dominey, P. F.; Hoen, M.; & Inui, T. (2006b). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088-2107.
33. Dou, D.; Frishkoff, G.; Rong, J.; Frank, R.; Malony, A.; & Tucker, D. (2007). Development of NeuroElectroMagnetic Ontologies(NEMO): a framework for mining brainwave ontologies. New York: Assoc Computing Machinery.
34. Draper, B. A.; Collins, R. T.; Brolio, J.; Hanson, A. R.; & Riseman, E. M. (1989). The schema system. International Journal of Computer Vision, 2, 209-250.
35. Evans. A. C.; Collins, D. L.; Mills, S. R.; Brown, E. D.; Kelly, R. L.; Peters, T. M. (1993). 1813-17 vol.3-13 -17 vol.3.
36. Fellbaum, C. (2010). WordNet. In R. Poli, M. Healy, & A. Kameas (Eds.), Theory and applications of ontology: computer applications (pp. 231-43). Springer: Netherlands.
37. Ferreira, F. (2003). The misinterpretation of noncanonical sentences. Cognitive Psychology, 47, 164-203.
38. Ferreira, F.; & Patson, N. D. (2007). The 'good enough' approach to language comprehension. Language and Linguistics Compass, 1, 71-83.
39. Fox, P. T.; & Lancaster, J. L. (2002). Mapping context and content: the BrainMap model. Nature Reviews Neuroscience, 3, 319-321.
40. Fox, P. T.; Laird, A. R.; Fox, S. P.; Fox, P. M.; Uecker, A. M.; et al. (2005). Brainmap taxonomy of experimental design: description and evaluation. Human Brain Mapping, 25, 185-198.
41. Frazier, L.; & Fodor, J. D. (1978). The sausage machine: a new two-stage parsing model. Cognition, 6, 291-325.
42. Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78-84.
43. Friederici, A. D. (2009). Pathways to language: fiber tracts in the human brain. Trends in Cognitive Sciences, 13, 175-181.
44. Gleitman, L. R.; January, D.; Nappa, R.; & Trueswell, J. C. (2007). On the give and take between even apprehension and utterance formulation. Journal of Memory and Language, 57, 544-569.
45. Goldberg, A. E. (1995). Constructions: a construction grammar approach to argument structure. Chicago: The University of Chicago Press.
46. Goodglass, H. (1976). Agrammatism. Studies in Neurolinguistics, 1, 237-260.
47. Grodzinsky, Y. (2000). The neurology of syntax: language use without Broca's area. The Behavioral and Brain Sciences, 23, 1-21.
48. Grodzinsky, Y.; Piñango, M. M.; Zurif, E.; & Drai, D. (1999). The critical role of group studies in neuropsychology: comprehension regularities in Broca's Aphasia. Brain and Language, 67, 134-147.
49. Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sciences, 9, 416-423.
50. Hanson, A. R.; & Riseman, E. M. (1978). VISIONS: a computer system for interpreting scenes. In A. R. Hanson & E. M. Riseman (Eds.), Computer vision systems (pp. 129-163). New York: Academic.
51. Hawkins, J. A. (1999). Processing complexity and Filler-Gap dependencies across grammars. Language & Cognitive Processes, 75, 244-285.
52. Hickok, G.; & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67-99.
53. Hurford, J. R. (2011). The origins of grammar II: language in the light of evolution. Oxford, New York: Oxford University Press.
54. Kay, P. (2002). An informal sketch of the formal architecture of construction grammar. Grammars, 5, 1-19.
55. Kay, P.; Fillmore, C. J. (1999). Grammatical constructions and linguistic generalizations: the what's X doing Y? Construction.
56. Kemmerer, D. (2000a). Grammatically relevant and grammatically irrelevant features of verb meaning can be independently impaired. Aphasiology, 14, 997-1020.
57. Kemmerer, D. (2000b). Selective impairment of knowledge underlying prenominal adjective order: evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 13, 57-82.
58. Kemmerer, D. (2003). Why can you hit someone on the arm but not break someone on the arm? - a neuropsychological investigation of the English body-part possessor ascension construction. Journal of Neurolinguistics, 16, 13-36.
59. Kemmerer, D.; & Wright, S. K. (2002). Selective impairment of knowledge underlying un- prefixation: further evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 15, 403-432.
60. Kemmerer, D.; Tranel, D.; & Zdanczyk, C. (2009). Knowledge of the semantic constraints on adjective order can be selectively impaired. Journal of Neurolinguistics, 22, 91-108.
61. Kempen, G.; Olsthoorn, N.; & Sprenger, S. (2012). Grammatical workspace sharing during language production and language comprehension: evidence from grammatical multitasking. Language & Cognitive Processes, 27, 345-380.
62. Kim, & Osterhout. (2005). The independence of combinatory semantic processing: evidence from event-related potentials. Journal of Memory and Language, 52, 205-225.
63. Kos, M.; Vosse, T.; Dvd, B.; & Hagoort, P. (2010). About edible restaurants: conflicts between syntax and semantics as revealed by ERPs. Frontiers in Language Sciences, 1, 222-22.
64. Kudo, T. (1984). The effect of semantic plausibility on sentence comprehension in aphasia. Brain and Language, 21, 208-218.
65. Kuperberg, G. R. (2007). Neural mechanisms of language comprehension: challenges to syntax. Brain Research, 1146, 23-49.
66. Laird, A. R.; Eickhoff, S. B.; Kurth, F.; Fox, P. M.; Uecker, A. M.; et al. (2009). ALE meta-analysis workflows via the brainmap database: progress towards a probabilistic functional brain atlas. Frontiers in Neuroinformatics, 3, 23-23.
67. Lee J. (2012). Linking eyes to mouth: a schema-based computational model for describing visual scenes. Ph.D. Thesis, Computer Science, University of Southern California, Los Angeles, CA.
68. Lee J. (In preparation-a). Implementing Template Construction Grammar (TCG) for visual scene description.
69. Lee J. (In preparation-b). The temporal unfolding of eye movements and utterance formulation.
70. Lesser, V. R.; Fennel, R. D.; Erman, L. D.; & Reddy, D. R. (1975). Organization of the HEARSAY-II speech understanding system. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23, 11-23.
71. Letovsky, S. I.; Whitehead, S. H.; Paik, C. H.; Miller, G. A.; Gerber, J.; et al. (1998). A brain image database for structure/function analysis. AJNR. American Journal of Neuroradiology, 19, 1869-1877.
72. Levin B. (1993). English verb classes and alternations: a preliminary investigation. University of Chicago Press.
73. Luria, A. R. (1973). The working brain. Harmondsworth: Penguin.
74. MacWhinney, B. (2007). The TalkBank Project.
75. MacWhinney, B.; Fromm, D.; Forbes, M.; & Holland, A. (2011). AphasiaBank: methods for studying discourse. Aphasiology, 25, 1286-1307.
76. Makkai, A. (1972). Idiom structure in English
77. Marcus, D. S.; Harwell, J.; Olsen, T.; Hodge, M.; Glasser, M. F.; et al. (2011). Informatics and data mining tools and strategies for the human connectome project. Frontiers in Neuroinformatics, 5.
78. Mayberry, M.; Crocker, M. W.; Knoeferle, P. (2006). A connectionist model of the coordinated interplay of scene, utterance, and world knowledge.
79. Menenti, L.; Gierhan, S. M. E.; Segaert, K.; & Hagoort, P. (2011). Shared language overlap and segregation of the neuronal infrastructure for speaking and listening revealed by functional MRI. Psychological Science, 22, 1173-1182.
80. Miyake, A.; Carpenter, P. A.; & Just, M. A. (1994). A capacity approach to syntactic comprehension disorders: making normal adults perform like aphasic patients. Cognitive Neuropsychology, 11, 671-717.
81. Miyake, A.; Carpenter, P. A.; & Just, M. A. (1995). Reduced resources and specific impairments in normal and aphasic sentence comprehension. Cognitive Neuropsychology, 12, 651-679.
82. Mohanan, T.; Wee, L. (1999). Grammatical semantics: evidence for structure in meaning. CSLI.
83. Osterhout, L.; Albert, K.; Kuperberg, G. (2007). The neurobiology of sentence comprehension, CiteSeerX.
84. Pinker, S. (1989). Learnability and cognition: the acquisition of argument structure. Cambridge: The MIT Press. 411 pp.
85. Poeppel, D.; Emmorey, K.; Hickok, G.; & Pylkkänen, L. (2012). Towards a new neurobiology of language. The Journal of Neuroscience, 32, 14125-14131.
86. Saffran, E. M.; Schwartz, M. F.; & Linebarger, M. C. (1998). Semantic influences on thematic role assignment: evidence from normals and aphasics. Brain and Language, 62, 255-297.
87. Schuler, K. K. (2005). VerbNet: a broad-coverage, comprehensive verb lexicon. University of Pennsylvania
88. Schwartz, M. F.; Linebarger, M. C.; Saffran, E. M.; & Pate, D. S. (1987). Syntactic transparency and sentence interpretation in aphasia. Language & Cognitive Processes, 2, 85-113.
89. Segaert, K.; Menenti, L.; Weber, K.; Petersson, K. M.; & Hagoort, P. (2012). Shared syntax in language production and language comprehension - an fMRI study. Cerebral Cortex, 22, 1662-1670.
90. Sherman, J. C.; & Schweickert, J. (1989). Syntactic and semantic contributions to sentence comprehension in agrammatism. Brain and Language, 37, 419-439.
91. Spivey, M. J.; Richardson, D. C.; & Fitneva, S. A. (2005). Thinking outside the brain: spatial indices to visual and linguistic information. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: eye movements and the visual world (pp. 161-190). New York, Hove: Psychology Press.
92. Sporns, O.; Tononi, G.; & Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Computational Biology, 1, e42-e42.
93. Steels, L. (1999). The talking heads experiment.
94. Steels, L.; & De Beule, J. (2006). Unify and merge in fluid construction grammar. In P. Vogt, Y. Sugita, E. Tuci, & C. Nehaniv (Eds.), Symbol grounding and beyond, proceedings (pp. 197-223). Berlin: Springer-Verlag Berlin.
95. Stefanowitsch, A.; & Gries, S. T. (2003). Collostructions: investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8, 209-243.
96. Stephan, K. E.; Kamper, L.; Bozkurt, A.; Burns, G. A. P. C.; Young, M. P.; & Kotter, R. (2001). Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). Philosophical Transactions of the Royal Society B: Biological Sciences, 356, 1159-1186.
97. Talairach, Tournoux P. (1988). Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: an approach to cerebral imaging. Thieme.
98. Tanenhaus, M. K.; Spivey-Knowlton, M. J.; Eberhard, K. M.; & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.
99. Van Essen, D. C. (2009). Lost in localization-but found with foci?! NeuroImage, 48, 14-17.
100. Vigneau, M.; Beaucousin, V.; Hervé, P. Y.; Duffau, H.; Crivello, F.; et al. (2006). Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. NeuroImage, 30, 1414-1432.
101. Vosse, T.; & Kempen, G. (2009). In defense of competition during syntactic ambiguity resolution. Journal of Psycholinguistic Research, 38, 1-9.
102. Wendel, K.; Väisänen, O.; Malmivuo. J.; Gencer, N. G.; Vanrumste. B.; et al. (2009). EEG/MEG source imaging: methods, challenges, and open issues. Intelligence and Neuroscience, 2009: 13:1-13:12-13:1-13:12.
103. Wilbur, R.; Kak, A. (2006). Purdue RVL-SLLL American Sign Language Database. ECE Technical Reports.
104. Zurif, E. B.; & Piñango, M. M. (1999). The existence of comprehension patterns in Broca's Aphasia. Brain and Language, 70, 133-138.