Using reinforcement learning to understand the emergence of "intelligent" eye-movement behavior during reading

Psychological Review

Volumn 113, Issue 2, 2006, Pages 390-408

  Reichle, Erik D ^a,b   Laurent, Patryk A ^a  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

Attention; Computational models; Eye movement control; Reading; Reinforcement learning

Indexed keywords

ALGORITHM; ARTICLE; ATTENTION; EYE FIXATION; EYE MOVEMENT; HUMAN; INTELLIGENCE; LEARNING; MONTE CARLO METHOD; MOTIVATION; PSYCHOPHYSICS; READING; REINFORCEMENT; SACCADIC EYE MOVEMENT; STATISTICAL MODEL;

ALGORITHMS; ASSOCIATION LEARNING; ATTENTION; EYE MOVEMENTS; FIXATION, OCULAR; HUMANS; INTELLIGENCE; MODELS, STATISTICAL; MONTE CARLO METHOD; MOTIVATION; PSYCHOPHYSICS; READING; REINFORCEMENT (PSYCHOLOGY); SACCADES; VERBAL LEARNING;

EID: 33744543495     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/0033-295X.113.2.390     Document Type: Article

References (106)

1. Altarriba, J., Kroll, J. F., Sholl, A., & Rayner, K. (1996). The influence of lexical and conceptual constraints on reading mixed language sentences: Evidence from eye fixation and naming times. Memory & Cognition, 24, 477-492.
2. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In A. Prieditis & S. J. Russell (Eds.), Machine learning: Proceedings of the Twelfth International Conference (pp. 30-37). San Francisco: Morgan Kaufman.
3. Balota, D. A., Pollatsek, A., & Rayner, K. (1985). The interaction of contextual constraints and parafovcal visual information in reading. Cognitive Psychology, 17, 364-390.
4. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. Davis, & D. Beiser (Eds.), Models of information processing in the basal ganglia (pp. 215-232). Cambridge, MA: MIT Press.
5. Becker, W., & Jürgens, R. (1979). An analysis of the saccadic system by means of double step stimuli. Vision Research, 19, 967-983.
6. Brysbaert. M., & Vitu, F. (1998). Word skipping: Implications for theories of eye movement control in reading. In G. Underwood (Ed.), Eye guidance in reading and scene perception (pp. 125-147). Amsterdam: Elsevier.
7. Clark, V. P., Fan, S., & Hillyard, S. A. (1995). Identification of early visual evoked potential generators by retinotopic and topographic analyses. Human Brain Mapping, 2, 170-187.
8. Coulom, R. (2002). Reinforcement learning using neural networks, with applications to motor control. Unpublished doctoral dissertation, Institut National Polytechnique de Grenoble, Grenoble, France.
9. Ehrlich, S. F., & Rayner, K. (1981). Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior, 20, 641-655.
10. Engbert, R., Longtin, A., & Kliegl, R. (2002). A dynamical model of saccade generation in reading based on spatially distributed lexical processing. Vision Research, 42, 621-636.
11. Engbert, R., Kliegl, R., & Longtin, A. (2004). Complexity of eye movements in reading. International Journal of Bifurcation and Chaos, 14, 493-503.
12. Engbert, R., Nuthmann, A., Richter, E., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112, 777-813.
13. Foxe, J. J., & Simpson, G. V. (2002). Flow of activation from V1 to frontal cortex in humans: A framework for defining "early" visual processing. Experimental Brain Research, 142, 139-150.
14. Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178-210.
15. Henderson, J. M., & Ferreira, F. (1990). Effects of foveal processing difficulty on the perceptual span in reading: Implications for attention and eye movement control. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16, 417-429.
16. Hikosaka, O., Takikawa, Y., & Kawagoe, R. (2000). Role of the basal ganglia in the control of purposive saccadic eye movements. Psychological Review, 80, 953-978.
17. Hyönä, J., & Olson, R. K. (1995). Eye movement patterns among dyslexic and normal readers: Effects of word length and word frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 1430-1440.
18. Inhoff, A. W., Eiter, B. M., & Radach, R. (in press). The time course of linguistic information extraction from consecutive words during eye fixations in reading. Journal of Experimental Psychology; Human Perception and Performance.
19. Inhoff, A. W., & Rayner, K. (1986). Parafoveal word processing during eyefixations in reading: Effects of word frequency. Perception & Psychophysics, 40, 431-439.
20. Inhoff, A. W., Stare, M., & Shindler, K. L. (2000). Is the processing of words during eye fixations in reading strictly serial? Perception & Psychophysics, 62, 1474-1484.
21. Ishida, T., & Ilceda, M. (1989). Temporal properties of information extraction in reading studied by a text-mask replacement technique. Journal of the Optical Society A: Optics and Image Sciences, 6, 1624-1632.
22. Juhasz, B., & Rayner, K. (2003). Investigating the effects of a set of intercorrelated variables on eye fixation durations during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6, 1312-1318.
23. Juhasz, B., & Rayner, K. (in press). The role of age-of-acquisition and word frequency in reading: Evidence from eye fixation durations. Visual Cognition.
24. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329-354.
25. Just, M. A., & Carpenter, P. A. (1987). The psychology of reading and language comprehension. Boston: Allyn & Bacon.
26. Kaiser, E., & Trueswell, J. (2004). The role of discourse context in the processing of a flexible word-order language. Cognition, 94, 113-147.
27. Kennedy, A. (1998). The influence of parafoveal words of foveal inspection time: Evidence for a processing trade-off. In G. Underwood (Ed.), Eye guidance in reading a scene perception (pp. 149-223). Amsterdam: Elsevier.
28. Kennedy, A. (2000). Parafoveal processing in word recognition. Quarterly Journal of Experimental Psychology, 53A, 429-455.
29. Kennedy, A., Murray, W. S., & Boissiere, C. (2004). Parafoveal pragmatics revisited. European Journal of Cognitive Psychology, 16, 128-153.
30. Kennedy, A., Pynte, J., & Ducrot, S. (2002). Parafoval-on-foveal interactions in word recognition. Quarterly Journal of Experimental Psychology, 55A, 1307-1337.
31. Kennison, S. M., & Clifton, C. (1995). Determinants of parafoveal preview benefit in high and low working memory capacity readers: Implications for eye movement control. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 68-81.
32. Kliegl, R., & Engbert, R. (2003). SWIFT explorations. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 391-411). Oxford, England: Elsevier.
33. Kliegl, R., & Engbert, R. (2005). Fixation durations before word skipping in reading. Psychonomic Bulletin & Review, 12, 132-138.
34. Kliegl, R., Grabner. E., Rolfs, M., & Engbert, R. (2004). Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology, 16, 262-284.
35. Klitz, T. S., Legge, G. E., & Tjan, B. S. (2000). Saccade planning in reading with central scotomas: Comparison of human and ideal performance. In A. Kennedy, R. Radach, D. Heller, & J. Pynte (Eds.), Reading as a perceptual process (pp. 667-682). Oxford, England: Elsevier.
36. Lee, H.-W., Legge, G. E., & Ortiz, A. (2003). Is word recognition different in central and peripheral vision? Vision Research, 43, 2837-2846.
37. Leff, A. P., Scott, S. K., Rothwell, J. C., & Wise, R. J. S. (2001). The planning and guiding of reading saccades: A repetitive transcranial magnetic stimulation study. Cerebral Cortex, 11, 918-923.
38. Legge, G. E., Hooven, T. A., Klitz, T. S., Mansfield, J. S., & Tjan, B. S. (2002). Mr. Chips 2002: New insights for an ideal-observer model of reading. Vision Research, 42, 2219-2234.
39. Legge, G. E., Klitz, T. S., & Tjan, B. S. (1997). Mr. Chips: An idealobserver model of reading. Psychological Review, 104, 524-553.
40. McConkie, G. W., Kerr, P. W., & Dyre, B. P. (1994). What are "normal" eye movements during reading: Toward a mathematical description. In J. Ygge & G. Lennerstrand (Eds.), Eye movements in reading (pp. 315-328). New York: Pergamon Press.
41. McConkie, G. W., Kerr, P. W., Reddix, M. D., & Zola, D. (1988). Eye movement control during reading: I. The location of initial eye fixations in words. Vision Research, 28, 1107-1118.
42. McConkie, G. W., Zola, D., Grimes, J., Kerr, P. W., Bryant, N. R., & Wolff, P. M. (1991). Children's eye movements during reading. In J. F. Stein (Ed.), Vision and visual dyslexia 13. London: MacMillan.
43. McDonald, S. A., Carpenter, R. H. S., & Shillcock, R. C. (2005). An anatomically constrained, stochastic model of eye movement control in reading. Psychological Review, 112, 814-840.
44. McPeek, R. M., Skavenski, A. A., & Nakayama, K. (2000). Concurrent processing of saccades in visual search. Vision Research, 40, 2499-2516.
45. Molker, A., & Fischer, B. (1999). The recognition and correction of involuntary prosaccades in an antisaccade task. Experimental Brain Research, 125, 511-516.
46. Mouchetant-Rostaing, Y., Giard, M.-H., Bentin, S., Aguera, P.-E., & Pernier, J. (2000). Neurophysiological correlates of face gender processing in humans. European Journal of Neuroscience, 12, 303-310.
47. Murray, W. S. (1998). Parafoval pragmatics. In G. Underwood (Ed.), Eye guidance in reading a scene perception (pp. 181-199). Amsterdam: Elsevier.
48. Nuthmann, A., Engbert, R., & Kliegl, R. (2005). Mislocated fixations during reading and the inverted optimal viewing position effect. Vision Research, 45, 2201-2217.
49. O'Regan, J. K. (1990). Eye movements and reading. In E. Kowler (Ed.), Eye movements and their role in visual and cognitive processes (pp. 395-453). Amsterdam: Elsevier.
50. O'Regan, J. K. (1992). Optimal viewing position in words and the strategytactics theory of eye movements in reading. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 333-354). Springer-Verlag.
51. O'Regan, J. K., & Lévy-Schoen, A. (1987). Eye movement strategy and tactics in word recognition and reading. In M. Coltheart (FM.), Attention and performance XII: The psychology of reading (pp. 363-383). Hillsdale, NJ: Erlbaum.
52. O'Regan, J. K., Lévy-Schoen, A., Pynte, J., & Brugaillère, B. (1984). Convenient fixation location within isolated words of different length and structure. Journal of Experimental Psychology: Human Perception and Performance, 10, 250-257.
53. Perfetti, C. A. (1985). Reading ability. New York: Oxford University Press.
54. Perfetti, C. A., & Hart, L. (2002). The lexical quality hypothesis. In L. Verhoeven, C. Elbro, & P. Reitsma (Eds.), Precursors of functional literacy (pp. 189-213). Philadelphia: John Benjamins.
55. Pollatsek, A., & Rayner, K. (1999). Is covert attention really unnecessary? Behavioral and Brain Sciences, 22, 695-696.
56. Pollatsek, A., Rayner, K., & Balota, D. A. (1986). Inferences about eye movement control from the perceptual span in reading. Perception & Psychophysics, 40, 123-130.
57. Pollatsek, A., Reichle, E. D., & Rayner, K. (2006). Tests of the E-Z Reader model: Exploring the interface between cognition and eye-movement control. Cognitive Psychology, 52, 1-56.
58. Pynte, J., Kennedy, A., & Ducrot, S. (2004). The influence of parafoveal typographical errors on eye movements in reading. European Journal of Cognitive Psychology, 16, 178-202.
59. Radach, R., & Heller, D. (2000). Relations between spatial and temporal aspects of eye movement control. In A. Kennedy, R. Radach, D. Heller, & J. Pynte (Eds.). Reading as a perceptual process (pp. 165-192). Oxford, England: Elsevier.
60. Rayner, K. (1977). Visual attention in reading: Eye movements reflect cognitive processes. Memory & Cognition, 4, 443-448.
61. Rayner, K. (1978). Eye movements in reading and information processing. Psychological Bulletin, 85, 618-660.
62. Rayner, K. (1979). Eye guidance in reading: Fixation locations within words. Perception, 8, 21-30.
63. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.
64. Rayner, K., Ashby, J., Pollatsek, A., & Reichle, E. (2004). The effects of frequency and predictability on eye fixations in reading: Implications for the E-Z Reader model. Journal of Experimental Psychology: Human Perception and Performance, 30, 720-732.
65. Rayner, K., & Bertera, J. H. (1979). Reading without a fovea. Science, 206, 468-469.
66. Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14, 191-201.
67. Rayner, K., & Juhasz, B. (2004). Eye movements in reading: Old questions and new directions. European Journal of Cognitive Psychology, 16, 340-352.
68. Rayner, K., & McConkie, G. W. (1976). What guides a reader's eye movements? Vision Research, 16, 829-837.
69. Rayner, K., & Morrison, R. M. (1981). Eye movements and identifying words in parafoveal vision. Bulletin of the Psychonomic Society, 17, 135-138.
70. Rayner, K., & Pollatsek, A. (1989). The psychology of reading. Englewood Cliffs, NJ: Prentice Hall.
71. Rayner, K., Pollatsek, A., & Reichle, E. D. (2003). Eye movements in reading: Models and data. Brain and Behavioral Sciences, 26, 507-526.
72. Rayner, K., & Raney, G. E. (1996). Eye movement control in reading and visual search: Effects of word frequency. Psychonomic Bulletin & Review, 3, 238-244.
73. Rayner, K., Sereno, S. C., & Raney, G. E. (1996). Eye movement control in reading: A comparison of two types of models. Journal of Experimental Psychology: Human Perception and Performance, 22, 1188-1200.
74. Rayner, K., Slowiaczek, M. L., Clifton, C., & Bertera, J. H. (1983). Latency of sequential eye movements: Implications for reading. Journal of Experimental Psychology: Human Perception and Performance, 9, 912-922.
75. Rayner, K., & Well, A. D. (1996). Effects of contextual constraint of eye movements in reading: A further examination. Psychonomic Bulletin & Review, 3, 504-509.
76. Rayner, K., White, S. J., Kambe, G., Miller, B., & Liversedge, S. P. (2003). On the processing of meaning from parafoveal vision during eye fixations in reading. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 213-234). Oxford, England: Elsevier.
77. Reichle, E. D., & Laurent, P. A. (2004, September). The emergence of "intelligent" eye-movement control during reading: A computational account. Poster session presented at Architectures and Mechanisms for Language Processing 2004. Aix-en-Provence, France.
78. Reichle, E. D., & Perfetti, C. A. (2003). Morphology in word identification: A word experience model that accounts for morpheme frequency effects. Scientific Studies of Reading, 7, 219-237.
79. Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Towards a model of eye movement control in reading. Psychological Review, 125-157.
80. Reichle, E. D., Rayner, K., & Pollatsek, A. (1999). Eye movement control in reading: Accounting for initial fixation locations and refixations within the E-Z Reader model. Vision Research, 39, 4403-4411.
81. Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The E-Z Reader model of eye movement control in reading: Comparison to other models. Brain and Behavioral Sciences, 26, 445-476.
82. Reichle, E. D., Rayner, K., & Pollatsek, A. (2006). E-Z Reader: A cognitive-control, serial-attention model of eye-movement control during reading. Cognitive Systems Research, 7, 4-22.
83. Reilly, R. (1993). A connectionist framework for model eye movement control in reading. In G. d'Ydewalle & J. Van Rensbergen (Eds.), Perception and cognition: Advances in eye movement research (pp. 193-212). Amsterdam: North-Holland.
84. Reilly, R., & O'Regan, J. K. (1998). Eye movement control in reading: A simulation of some word-targeting strategies. Vision Research, 38, 303-317.
85. Reilly, R., & Radach, R. (2003). Foundations of an interactive activation model of eye movement control in reading. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movement research (pp. 429-455). Oxford, England: Elsevier.
86. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, & the POP Research Group (Eds.), Parallel distributed processing: Explorations in the microstructure of cognition, Vol. 1: Foundations (pp. 318-362). Cambridge, MA: MIT Press.
87. Salvucci, D. D. (2000). An integrated model of eye movements and visual encoding. Cognitive Systems Research, 1, 201-220.
88. Schilling, H. E. H., Rayner, K., & Chumbley, J. I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory & Cognition, 26, 1270-1281.
89. Schullz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
90. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1598.
91. Sereno, S. C. (1992). Early lexical effects when fixating a word in reading. In K. Rayner (Ed.). Eye movements and visual cognition: Scene perception and reading (pp. 304-316). New York: Springer-Verlag.
92. Sereno, S. C., Rayner, K., & Posner, M. I. (1998). Establishing a time-line of word recognition: Evidence from eye movements and event-related potentials. NeuroReport, 9, 2195-3000.
93. Thibadeau, R., Just, M. A., & Carpenter, P. A. (1982). A model of the time course and content of reading. Cognitive Science, 6, 157-203.
94. Treisman, A., & Souther, J. (1986). Illusory words: The roles of attention and of top-down constraints in conjoining letters to form words. Journal of Experimental Psychology: Human Perception and. Performance, 12, 3-17.
95. Vallduvi, E., & Engdahl, E. (1996). The linguistic realization of information packaging. Linguistics, 34, 459-519.
96. Van Rullen, R., & Thorpe, S. (2001). The time course of visual processing: From early perception to decision-making. Journal of Cognitive Neuroscience, 13, 454-461.
97. Vergilino, D., & Beauvillain, C. (2000). The planning of refixation saccades in reading. Vision Research, 40, 3527-3538.
98. Vitu, F., Brysbaert, M., & Lancelin, D. (2004). A test of parafoveal-onfoveal effects with pairs of orthographically related words. European Journal of Cognitive Psychology, 16, 154-177.
99. Vitu, F., McConkie, G. W., Kerr, P., & O'Regan, J. K. (2001). Fixation location effects on fixation locations during reading: An inverted optimal viewing position effect. Vision Research, 41, 3513-3533.
100. Wheeler, M. E., & Treisman, A. M. (2002). Binding in short-term visual memory. Journal of Experimental Psychology: General, 131, 48-64.
101. Wolfe, J. M. (1994). Guided Search 2.0: A revised model of visual search. Psychonomic Bulletin & Review, 1, 202-238.
102. Wolfe, J. M., & Bennett, S. C. (1996). Preattentive object files: Shapeless bundles of basic features. Vision Research, 37, 25-43.
103. Wolverton, G. S., & Zola, D. (1983). The temporal characteristics of visual information extraction during reading. In K. Rayner (Ed.), Eye movements in reading: Perceptual and language processes (pp. 41-52). New York: Academic Press.
104. Wörgötter, F. (2004, February-March). Actor-critic model of animal control: A critique of reinforcement learning. Paper presented at the Fourth International ICSC Symposium on Engineering of Intelligent Systems, Madeira, Portugal.
105. Yang, S.-N., & McConkie, G. W. (2001). Eye movements during reading: A theory of saccade initiation time. Vision Research, 41, 3567-3568.
106. Yang, S.-N., & McConkie, G. W. (2004). Saccade generation during reading: Are words necessary? European Journal of Cognitive Psychology, 16, 226-261.