Memory and surprisal in human sentence comprehension

Sentence Processing

Volumn , Issue , 2013, Pages 78-114

  Levy, Roger ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84917470887     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.4324/9780203488454     Document Type: Chapter

References (137)

1. Adams, B. C., Clifton, Jr., C., & Mitchell, D. C. (1998). Lexical guidance in sentence processing? Psychonomic Bulletin & Review, 5(2), 265–270.
2. Altmann, G. T., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73(3),247–264.
3. Anderson, J. R. (1990). The adaptive character of thought. Hillsdale, NJ: Lawrence Erlbaum.
4. Arcuri, S. M., Rabe-Hesketh, S., Morris, R. G., & McGuire, P. K. (2001). Regional variation of Cloze probabilities for sentence contexts. Behavior Research Methods, Instruments, and Computers, 33(1), 80–90.
5. Attneave, F. (1959). Applications of information theory to psychology: A summary of basic concepts, methods and results. New York: Holt, Rinehart and Winston.
6. Bever, T. (1970). The cognitive basis for linguistic structures. In Hayes, J. (Ed.), Cognition and the development of language (pp. 279–362). Chichester: John Wiley & Sons.
7. Bicknell, K., & Levy, R. (2009). A model of local coherence effects in human sentence processing as consequences of updates from bottom-up prior to posterior beliefs. In Proceedings of the North American chapter of the Association for Computational Linguistics-Human Language Technologies (NAACL-HLT) conference.
8. Bicknell, K., & Levy, R. (2010). A rational model of eye-movement control in reading. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (pp. 1168–1178).
9. Bicknell, K., & Levy, R. (2012). Word predictability and frequency effects in a rational model of reading. In Proceedings of the 34th Annual Meeting of the Cognitive Science Conference.
10. Bock, K., & Miller, C. A. (1991). Broken agreement. Cognitive Psychology, 23, 45–93.
11. Booth, T. L. (1969). Probabilistic representation of formal languages. In IEEE Conference Record of the 1969 Tenth Annual Symposium on Switching and Automata Theory (pp. 74–81).
12. Boston, M. F., Hale, J. T., Kliegl, R., Patil, U., & Vasishth, S. (2008). Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam sentence corpus. Journal of Eye-Movement Research, 2(1), 1–12.
13. Charniak, E. (1996). Tree-bank grammars. Technical report, Department of Computer Science, Brown University.
14. Chen, E., Gibson, E., & Wolf, F. (2005). On-line syntactic storage costs in sentence comprehension. Journal of Memory and Language, 52, 144–169.
15. Chomsky, N. (1956). Three models for the description of language. Ire Transactions on Information Theory, 2(3), 113–124.
16. Chomsky, N. (1957). Syntactic structures. Oxford: Mouton.
17. Chomsky, N. (1965). Aspects of the theory of syntax. Cambridge, MA: Mit Press.
18. Chomsky, N. (1981). Lectures on government and binding. Berlin and New York: Mouton de Gruyer.
19. Chomsky, N., & Miller, G. A. (1963). Introduction to the formal analysis of natural languages. In R. D. Luce, R. R. Bush, & E. Galanter (Eds.), Handbook of mathematical psychology, Vol. Ii (pp. 269–321). Chichester: John Wiley & Sons.
20. Christiansen, M. H., & MacDonald, M. C. (2009). A usage-based approach to recursion in sentence processing. Language Learning, 51(1), 126–161.
21. Clifton, C. Jr. (1993). Thematic roles in sentence parsing. Canadian Journal of Experimental Psychology, 47(2), 224–246.
22. Cover, T., & Thomas, J. (1991). Elements of information theory. Chichester: John Wiley.
23. Cowper, E. A. (1976). Constraints on sentence complexity: A model for syntactic processing. Ph.D. Thesis, Brown University, Providence, RI.
24. Crocker, M., & Brants, T. (2000). Wide-coverage probabilistic sentence processing. Journal of Psycholinguistic Research, 29(6), 647–669.
25. DeLong, K. A., Urbach, T. P., & Kutas, M. (2005). Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience, 8, 1117–1121.
26. Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2), 193–210.
27. Demberg, V., & Keller, F. (2009). A computational model of prediction in human parsing: Unifying locality and surprisal effects. In Proceedings of CogSci.
28. Eberhard, K. M. (1999). The accessibility of conceptual number to the processes of subject-verb agreement in English. Journal of Memory and Language, 41, 560–578.
29. Eberhard, K. M., Cutting, J. C., & Bock, K. (2005). Making syntax of sense: Number agreement in sentence production. Psychological Review, 112(3),531–559.
30. Ehrlich, S. F., & Rayner, K. (1981). Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior, 20, 641–655.
31. Elman, J. (1990). Finding structure in time. Cognitive Science, 14, 179–211.
32. Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). Swift: A dynamical model of saccade generation during reading. Psychological Review, 112(4), 777–813.
33. Ferreira, F., & Henderson, J. (1990). Use of verb information in syntactic parsing: Evidence from eye-movements and word-by-word self-paced reading. Journal of Experimental Psychology: Learning, Memory, & Cognition, 16(4),555–568.
34. Ford, M. (1983). A method for obtaining measures of local parsing complexity throughout sentences. Journal of Verbal Learning and Verbal Behavior, 22, 203–218.
35. Fossum, V., & Levy, R. (2012). Sequential versus hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd annual workshop on cognitive modeling and computational linguistics.
36. Franck, J., Vigliocco, G., & Nicol, J. (2002). Subject-verb agreement errors in French and English: The role of syntactic hierarchy. Language & Cognitive Processes, 17(4), 371–404.
37. Frank, S. L. (2009). Surprisal-based comparison between a symbolic and a connectionist model of sentence processing. In Proceedings of the 31st annual conference of the Cognitive Science Society (pp. 1139–1144).
38. Frank, S. L., & Bod, R. (2011). Insensitivity of the human sentence-processing system to hierarchical structure. Psychological Science, 22(6), 829–834.
39. Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye-movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178–210.
40. Gibson, E. (1991). A computational theory of human linguistic processing: Memory limitations and processing breakdown. Ph.D. Thesis, Carnegie Mellon.
41. Gibson, E. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68, 1–76.
42. Gibson, E. (2000). The dependency locality theory: A distance-based theory of linguistic complexity. In A. Marantz, Y. Miyashita, & W. O’Neil (Eds.), Image, language, brain (pp. 95–126). Cambridge, MA: Mit Press.
43. Gibson, E., Desmet, T., Grodner, D., Watson, D., & Ko, K. (2005). Reading relative clauses in English. Language & Cognitive Processes, 16(2), 313–353.
44. Gibson, E., & Thomas, J. (1997). Processing load judgements in English: Evidence for the Syntactic Prediction Locality Theory of syntactic complexity. Manuscript, MIT, Cambridge, MA.
45. Gibson, E., & Thomas, J. (1999). The perception of complex ungrammatical sentences as grammatical. Language & Cognitive Processes, 14(3), 225–248.
46. Gildea, D., & Temperley, D. (2007). Optimizing grammars for minimum dependency length. In Proceedings of ACL.
47. Gildea, D., & Temperley, D. (2010). Do grammars minimize dependency length? Cognitive Science, 34, 286–310.
48. Gordon, P. C., Hendrick, R., & Johnson, M. (2001). Memory interference during language processing. Journal of Experimental Psychology: Learning, Memory, & Cognition, 27, 1411–1423.
49. Gordon, P. C., Hendrick, R., & Johnson, M. (2004). Effects of noun phrase type on sentence complexity. Journal of Memory and Language, 51(1), 97–114.
50. Grodner, D., Comer, K., & Gibson, E. (2011). Non-local syntactic influences in structural ambiguity resolution. In preparation.
51. Grodner, D., & Gibson, E. (2005). Some consequences of the serial nature of linguistic input. Cognitive Science, 29(2), 261–290.
52. Grodner, D., Gibson, E., & Tunstall, S. (2002). Syntactic complexity in ambiguity resolution. Journal of Memory and Language, 46, 267–295.
53. Hale, J. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the second meeting of the North American chapter of the Association for Computational Linguistics (pp. 159–166).
54. Holmes, V. M., & O’Regan, J. K. (1981). Eye fixation patterns during the reading of relative-clause sentences. Journal of Verbal Learning and Verbal Behavior, 20, 417–430.
55. Hsiao, F., & Gibson, E. (2003). Processing relative clauses in Chinese. Cognition, 90(11),3–27.
56. Humboldt, W. (1988/1836). On language. Cambridge University Press. Translated from the German by Peter Heath. Originally published as Uber die Verschienheit des Menschlichen Sprachbaues, 1836, Berlin.
57. Jelinek, F., & Lafferty, J. D. (1991). Computation of the probability of initial substring generation by stochastic context free grammars. Computational Linguistics, 17(3), 315–323.
58. Jurafsky, D. (1996). A probabilistic model of lexical and syntactic access and disambiguation. Cognitive Science, 20(2), 137–194.
59. Jurafsky, D., & Martin, J. H. (2008). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Second Edition. Englewood Cliffs, NJ: Prentice-Hall.
60. Kimball, J., & Aissen, J. (1971). I think, you think, he think. Linguistic Inquiry, 2, 241–246.
61. King, J., & Just, M. A. (1991). Individual differences in syntactic processing: The role of working memory. Journal of Memory and Language, 30(5), 580–602.
62. Konieczny, L. (1996). Human sentence processing: A semantics-oriented parsing approach. Ph.D. Thesis, Universitat Freiburg.
63. Konieczny, L. (2000). Locality and parsing complexity. Journal of Psycholinguistic Research, 29(6),627–645.
64. Konieczny, L., & Doring, P. (2003). Anticipation of clause-final heads: Evidence from eye-tracking and SRNs. In Proceedings of ICCS/ASCS.
65. Krylova, O., & Khavronina, S. (1988). Word order in Russian. Moscow: Russky Yazyk Publishers.
66. Kucera, H., & Francis, W. N. (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.
67. Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205.
68. Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161–163.
69. Laming, D. R. J. (1968). Information theory of choice-reaction times. Academic Press.
70. Levy, R. (2008A). Expectation-based syntactic comprehension. Cognition, 106, 1126–1177.
71. Levy, R. (2008B). A noisy-channel model of rational human sentence comprehension under uncertain input. In Proceedings of the 13th conference on empirical methods in natural language processing (pp. 234–243).
72. Levy, R., Bicknell, K., Slattery, T., & Rayner, K. (2009). Eye-movement evidence that readers maintain and act on uncertainty about past linguistic input. In Proceedings of the National Academy of Sciences, 106(50), 21086–21090.
73. Levy, R., Fedorenko, E., & Gibson, E. (2013). The syntactic complexity of Russian relative clauses. Under review.
74. Levy, R., & Keller, F. (2013). Expectation and locality effects in German verb-final structures. Journal of Memory and Language, 68(2), 199–222.
75. Lewis, R. L., & Vasishth, S. (2005). An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science, 29, 1–45.
76. Lewis, R. L., Vasishth, S., & Van Dyke, J. (2006). Computational principles of working memory in sentence comprehension. Trends in Cognitive Science, 10(10), 447–454.
77. Luce, R. D., Bush, R. R., & Galanter, E., editors (1963). Handbook of mathematical psychology, Vol. Ii. Chichester: John Wiley & Sons.
78. MacDonald, M. C. (1993). The interaction of lexical and syntactic ambiguity. Journal of Memory and Language, 32, 692–715.
79. Manning, C. D., & Schutze, H. (1999). Foundations of statistical natural language processing. Cambridge, MA: Mit Press.
80. Marcus, M. P., Santorini, B., & Marcinkiewicz, M. A. (1994). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330.
81. Marks, L., & Miller, G. A. (1964). The role of semantic and syntactic constraints in the memorization of English sentences. Journal of Verbal Learning and Verbal Behavior, 3, 1–5.
82. Marslen-Wilson, W. (1975). Sentence perception as an interactive parallel process. Science, 189(4198), 226–228.
83. McDonald, S. A., & Shillcock, R. C. (2003). Low-level predictive inference in reading: The influence of transitional probabilities on eye-movements. Vision Research, 43, 1735–1751.
84. McRae, K., Spivey-Knowlton, M. J., & Tanenhaus, M. K. (1998). Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38(3), 283–312.
85. Miller, G. A., & Chomsky, N. (1963). Finitary models of language users. In R. D. Luce, R. R. Bush, & E. Galanter (Eds.), Handbook of mathematical psychology, Vol. Ii (pp. 419–491). Chichester: John Wiley & Sons.
86. Miller, G. A., & Isard, S. (1963). Some perceptual consequences of linguistic rules. Journal of Verbal Learning and Verbal Behavior, 2, 217–228.
87. Mitchell, D. C. (1984). An evaluation of subject-paced reading tasks and other methods for investigating immediate processes in reading. In D. Kieras & M. A. Just (Eds.), New methods in reading comprehension. Hillsdale, NJ: Erlbaum.
88. Mitchell, D. C. (1987). Lexical guidance in human parsing: Locus and processing characteristics. In M. Coltheart (Ed.), Attention and Performance XII: The psychology of reading (pp. 601–618). London: Erlbaum.
89. Mitchell, D. C., Cuetos, F., Corley, M., & Brysbaert, M. (1995). Exposure-based models of human parsing: Evidence for the use of coarse-grained (nonlexical) statistical records. Journal of Psycholinguistic Research, 24, 469–488.
90. Monsalve, I. F., Frank, S. L., & Vigliocco, G. (2012). Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th conference of the European chapter of the Association for Computational Linguistics.
91. Nakatani, K., & Gibson, E. (2008). Distinguishing theories of syntactic expectation cost in sentence comprehension: Evidence from Japanese. Linguistics, 46(1), 63–87.
92. Nakatani, K., & Gibson, E. (2010). An on-line study of Japanese nesting complexity. Cognitive Science, 34(1), 94–112.
93. Narayanan, S., & Jurafsky, D. (1998). Bayesian models of human sentence processing. In Proceedings of the twelfth annual meeting of the Cognitive Science Society.
94. Narayanan, S., & Jurafsky, D. (2002). A Bayesian model predicts human parse preference and reading time in sentence processing. Advances in Neural Information Processing Systems, 14, 59–65.
95. Norris, D. (2006). The Bayesian Reader: Explaining word recognition as an optimal Bayesian decision process. Psychological Review, 113(2), 327–357.
96. Norris, D. (2009). Putting it all together: A unified account of word recognition and reaction-time distributions. Psychological Review, 116(1), 207–219.
97. Park, Y. A., & Levy, R. (2009). Minimal-length linearizations for mildly context-sensitive dependency trees. In Proceedings of the North American chapter of the Association for Computational Linguistics-Human Language Technologies (NAACL-HLT) conference.
98. Pearlmutter, N., Garnsey, S., & Bock, K. (1999). Agreement processes in sentence comprehension. Journal of Memory and Language, 41, 427–456.
99. Pickering, M. J., & Traxler, M. J. (1998). Plausibility and recovery from garden paths: An eye-tracking study. Journal of Experimental Psychology: Learning, Memory, & Cognition, 24(4), 940–961.
100. Pollard, C., & Sag, I. (1994). Head-driven phrase structure grammar. Chicago, IL: University of Chicago Press and Stanford, CA: Csli Publications.
101. Rayner, K. (1998). Eye-movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3),372–422.
102. Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Toward a model of eye-movement control in reading. Psychological Review, 105(1), 125–157.
103. Reilly, R. G., & Radach, R. (2006). Some empirical tests of an interactive activation model of eye-movement control in reading. Cognitive Systems Research, 7, 34–55.
104. Roark, B. (2001). Probabilistic top-down parsing and language modeling. Computational Linguistics, 27(2), 249–276.
105. Roark, B. (2004). Robust garden path parsing. Natural Language Engineering, 10(1), 1–24.
106. Roark, B., Bachrach, A., Cardenas, C., & Pallier, C. (2009). Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of EMNLP.
107. Shepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237(4820), 1317–1323.
108. Smith, N. J., & Levy, R. (2008). Optimal processing times in reading: A formal model and empirical investigation. In Proceedings of the 30th annual meeting of the Cognitive Science Society.
109. Smith, N. J., & Levy, R. (In press). The effect of word predictability on reading time is logarithmic. Manuscript, UC San Diego.
110. Spivey, M. J., & Tanenhaus, M. K. (1998). Syntactic ambiguity resolution in discourse: Modeling the effects of referential content and lexical frequency. Journal of Experimental Psychology: Learning, Memory, & Cognition, 24(6),1521–1543.
111. Staub, A. (2007). The parser doesn’t ignore intransitivity, after all. Journal of Experimental Psychology: Learning, Memory, & Cognition, 33(3), 550–569.
112. Staub, A. (2010). Eye-movements and processing difficulty in object relative clauses. Cognition, 116, 71–86.
113. Stolcke, A. (1995). An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Computational Linguistics, 21(2), 165–201.
114. Stone, M. (1960). Models for choice-reaction times. Psychometrika, 25(3), 251–260.
115. Sturt, P., Pickering, M. J., & Crocker, M. W. (1999). Structural change and reanalysis difficulty in language comprehension. Journal of Memory and Language, 40, 136–150.
116. Tabor, W., Galantucci, B., & Richardson, D. (2004). Effects of merely local syntactic coherence on sentence processing. Journal of Memory and Language, 50(4), 355–370.
117. Tabor, W., & Tanenhaus, M. K. (1999). Dynamical models of sentence processing. Cognitive Science, 23(4), 491–515.
118. Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.
119. Taylor, W. L. (1953). A new tool for measuring readability. Journalism Quarterly, 30, 415.
120. Tenenbaum, J. B., & Griffiths, T. L. (2001). Generalization, similarity, and Bayesian inference. Behavioral & Brain Sciences, 24, 629–640.
121. Traxler, M. J., Morris, R. K., & Seely, R. E. (2002). Processing subject and object relative clauses: Evidence from eye-movements. Journal of Memory and Language, 47, 69–90.
122. Traxler, M. J., Pickering, M. J., & Clifton, C. (1998). Adjunct attachment is not a form of lexical ambiguity resolution. Journal of Memory and Language, 39, 558–592.
123. Trueswell, J. C., Tanenhaus, M. K., & Garnsey, S. M. (1994). Semantic influences on parsing: Use of thematic role information in syntactic ambiguity resolution. Journal of Memory and Language, 33, 285–318.
124. Van Berkum, J. J. A., Brown, C. M., Zwitserlood, P., Kooijman, V., & Hagoort, P. (2005). Anticipating upcoming words in discourse: Evidence from ERPs and reading times. Journal of Experimental Psychology: Learning, Memory, & Cognition, 31(3), 443–467.
125. Van Dyke, J. A. (2007). Interference effects from grammatically unavailable constituents during sentence processing. Journal of Experimental Psychology: Learning, Memory, & Cognition, 33(2), 407–430.
126. Van Dyke, J. A., & Lewis, R. L. (2003). Distinguishing effects of structure and decay on attachment and repair: A retrieval interference theory of recovery from misanalyzed ambiguities. Journal of Memory and Language, 49(3), 285–316.
127. Van Gompel, R. P. G., & Pickering, M. J. (2001). Lexical guidance in sentence processing: A note on Adams, Clifton, and Mitchell (1998). Psychonomic Bulletin & Review, 8(4), 851–857.
128. Van Gompel, R. P. G., Pickering, M. J., Pearson, J., & Liversedge, S. P. (2005). Evidence against competition during syntactic ambiguity resolution. Journal of Memory and Language, 52, 284–307.
129. Van Gompel, R. P. G., Pickering, M. J., & Traxler, M. J. (2001). Reanalysis in sentence processing: Evidence against current constraint-based and two-stage models. Journal of Memory and Language, 45, 225–258.
130. Vasishth, S., & Lewis, R. L. (2006). Argument-head distance and processing complexity: Explaining both locality and anti-locality effects. Language, 82(4), 767–794.
131. Vasishth, S., Suckow, K., Lewis, R. L., & Kern, S. (2010). Short-term forgetting in sentence comprehension: Crosslinguistic evidence from verb-final structures. Language & Cognitive Processes, 25(4), 533–567.
132. Vigliocco, G., & Nicol, J. (1998). Separating hierarchical relations and word order in language production: Is proximity concord syntactic or linear? Cognition, 68(1), B13–29.
133. Wagers, M. W., Lau, E. F., & Phillips, C. (2009). Agreement attraction in comprehension: Representations and processes. Journal of Memory and Language, 61, 206–237.
134. Wanner, E., & Maratsos, M. (1978). An ATN approach to comprehension. In M. Halle, J. Bresnan, & G. A. Miller (Eds.), Linguistic theory and psychological reality. Cambridge, MA: Mit Press.
135. Warren, T., & Gibson, E. (2002). The influence of referential processing on sentence complexity. Cognition, 85(1), 79–112.
136. Wicha, N. Y. Y., Moreno, E. M., & Kutas, M. (2004). Anticipating words and their gender: An event-related brain potential study of semantic integration, gender expectancy, and gender agreement in Spanish sentence reading. Journal of Cognitive Neuroscience, 16(7), 1272–1288.
137. Yngve, V. (1960). A model and and hypothesis for language structure. In Proceedings of the American Philosophical Society (pp. 444–466).