Using Reinforcement Learning to Examine Dynamic Attention Allocation During Reading

Cognitive Science

Volumn 37, Issue 8, 2013, Pages 1507-1540

  Liu, Yanping ^a   Reichle, Erik D ^b   Gao, Ding Guo ^a  

^a SUN YAT SEN UNIVERSITY   (China)

^b UNIVERSITY OF PITTSBURGH SCHOOL OF MEDICINE   (United States)

Author keywords

Attention; Eye movements; Neural networks; Reading; Reinforcement learning

Indexed keywords

ARTICLE; ATTENTION; COMPUTER SIMULATION; EYE MOVEMENT; NEURAL NETWORKS; REACTION TIME; READING; REINFORCEMENT; REINFORCEMENT LEARNING;

ATTENTION; EYE MOVEMENTS; NEURAL NETWORKS; READING; REINFORCEMENT LEARNING;

ATTENTION; COMPUTER SIMULATION; EYE MOVEMENTS; REACTION TIME; READING; REINFORCEMENT (PSYCHOLOGY);

EID: 84887512804     PISSN: 03640213     EISSN: None     Source Type: Journal    
DOI: 10.1111/cogs.12027     Document Type: Article

References (98)

1. Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In A. Prieditis, & S. Russell (Eds.), Machine Learning: Proceedings of the Twelfth International Conference on Machine Learning (ICML95) (pp. 30-37). San Mateo, CA: Morgan Kaufman.
2. Balota, D. A., Pollatsek, A., & Rayner, K. (1985). The interaction of contextual constraints and parafoveal visual information in reading. Cognitive Psychology, 17, 364-390.
3. Becker, W., & Jürgens, R. (1979). An analysis of the saccadic system by means of double step stimuli. Vision Research, 19, 967-983.
4. Brysbaert, M., & Vitu, F. (1998). Word skipping: Implication for theories of eye movement control in reading. In G. Underwood (Ed.), Eye guidance in reading and scene perception (pp. 125-147). Oxford, England: Elsevier.
5. Clark, V. P., Fan, S., & Hillyard, S. A. (1995). Identification of early visual evoked potential generators by retinotopic and topographic analyses. Human Brain Mapping, 2, 170-187.
6. Clifton, C., Staub, A., & Rayner, K. (2007). Eye movements in reading words and sentences. In R. P. G. van Gompel, M. H. Fischer, W. S. Murray, & R. L. Hill (Eds.), Eye movements: A window on mind and brain (pp. 341-371). New York: Elsevier.
7. Drieghe, D. (2011). Parafoveal-on-foveal effects on eye movements during reading. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), Oxford handbook on eye movements (pp. 839-855). Oxford, England: Oxford University Press.
8. Drieghe, D., Rayner, K., & Pollatsek, A. (2007). Mislocated fixation can account for the parafoveal-on-foveal effects in eye movements during reading. Quarterly Journal of Experimental Psychology, 61, 1236-1349.
9. Engbert, R., & Kliegl, R. (2011). Parallel graded attention models of reading. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), Oxford handbook on eye movements (pp. 787-800). Oxford, England: Oxford University Press.
10. Engbert, R., Longtin, A., & Kliegl, R. (2002). A dynamical model of saccade generation in reading based on spatially distributed lexical processing. Vision Research, 42, 621-636.
11. Engbert, R., Nuthmann, A., Richter, E., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112, 777-813.
12. Erhlich, S. F., & Rayner, K. (1981). Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior, 20, 641-655.
13. Feng, G. (2006). Eye movements as time-series random variables: A stochastic model of eye movement control in reading. Cognitive Systems Research, 7, 70-95.
14. Fernando, C., Goldstein, R., & Szathmáry, E. (2010). The neuronal replicator hypothesis. Neural Computation, 22, 2809-2857.
15. Foxe, J. J., & Simpson, G. V. (2002). Flow of activation from V1 to frontal cortex in humans: A framework for defining "early" visual processing. Experimental Brain Research, 142, 139-150.
16. Fuchs, A. F. (1971). The saccadic system. In P. Bach-y-Rita, C. C. Collins, & J. E. Hyde (Eds.), The control of eye movements (pp. 343-362). New York: Academic Press.
17. Häikiö, T., Bertram, R., Hyönä, J., & Niemi, P. (2009). Development of the letter identity span in reading: Evidence from the eye movement moving window paradigm. Journal of Experimental Child Psychology, 102, 167-181.
18. Hansen, N. (2006). The CMA evolution strategy: A comparing review. In J. A. Lozano, P. Larrañaga, I. Inza, & E. Bengoetxea (Eds.), Towards a new evolutionary computation. Advances in estimation of distribution algorithms (Vol. 192, pp. 75-102). Berlin: Springer.
19. Hansen, N., & Kern, S. (2004). Evaluating the CMA evolution strategy on multimodal test functions. In X. Yao, E. Burke, J. A. Lozano, J. Smith, J. J. Merelo-Guervós, J. A. Bullianaria, J. Rowe, P. Tino, A. Kabán, & H.-P. Schwefel (Eds.), Eighth International Conference on Parallel Problem Solving from Nature (PPSV VIII). Vol. 3242 (pp. 282-291). Berlin: Springer.
20. Hansen, N., Müller, S. D., & Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11, 1-18.
21. Hansen, N., & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9, 159-195.
22. Igel, C., Hansen, N., & Roth, S. (2007). Covariance matrix adaptation for multi-objective optimization. Evolutionary Computation, 15, 1-28.
23. Inhoff, A. W., Eiter, B. M., & Radach, R. (2005). Time course of linguistic information extraction from consecutive words during eye fixations in reading. Journal of Experimental Psychology: Human Perception and Performance, 31, 979-995.
24. Inhoff, A. W., & Radach, R. (1998). Definition and computation of oculomotor measures in the study of cognitive processes. In G. Underwood (Ed.), Eye guidance in reading, driving and scene perception (pp. 29-53). New York: Elsevier.
25. Inhoff, A. W., Radach, R., & Eiter, B. (2006). Temporal overlap in the linguistic processing of successive words in reading: Reply to Pollatsek, Reichle, and Rayner (2006). Journal of Experimental Psychology: Human Perception and Performance, 32, 1490-1495.
26. Inhoff, A. W., & Rayner, K. (1986). Parafoveal word processing during eye fixations in reading: Effects of word frequency. Perception & Psychophysics, 40, 431-439.
27. Joseph, H. S. S. L., Liversedge, S. P., Blythe, H. I., White, S. J., & Rayner, K. (2009). Word length and landing position effects during reading in children and adults. Vision Research, 49, 2078-2086.
28. Juhasz, B. J., & Rayner, K. (2006). The role of age of acquisition and word frequency in reading: Evidence from eye fixation durations. Visual Cognition, 13, 846-863.
29. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 329-354.
30. Kennedy, A., & Pynte, J. (2005). Parafoveal-on-foveal effects in normal reading. Vision Research, 45, 153-168.
31. Kliegl, R., & Engbert, R. (2005). Fixation durations before work skipping in reading. Psychonomic Bulletin & Review, 12, 132-138.
32. Kliegl, R., Nuthmann, A., & Engbert, R. (2006). Tracking the mind during reading: The influence of past, present, and future words on fixation durations. Journal of Experimental Psychology: General, 135, 12-35.
33. Li, X., Liu, P., & Rayner, K. (2011). Eye movement guidance in Chinese reading: Is there a preferred viewing location? Vision Research, 51, 1146-1156.
34. Liu, Y., & Reichle, E. D. (2010). The emergence of adaptive eye movements in reading. In S. Ohlsson & R. Catrabone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 1136-1141). Austin, TX: Cognitive Science Society.
35. Matin, E. (1974). Saccadic suppression: A review. Psychological Bulletin, 81, 899-917.
36. McConkie, G. W., Kerr, P. W., & Dyre, B. P. (1994). What are "normal" eye movements during reading: Toward a mathematical description. In J. Ygge & G. Lennerstand (Eds.), Eye movements in reading (pp. 315-327). New York: Pergamon Press.
37. McConkie, G. W., Kerr, P. W., Reddix, M. D., & Zola, D. (1988). Eye movement control during reading: I. The location of initial eye fixations in words. Vision Research, 28, 1107-1118.
38. McConkie, G. W., Kerr, P. W., Reddix, M. D., Zola, D., & Jacobs, A. M. (1989). Eye movement control during reading: II. Frequency of refixating a word. Perception & Psychophysics, 46, 245-253.
39. McConkie, G. W., & Rayner, K. (1975). The span of effective stimulus during a fixation in reading. Perception & Psychophysics, 17, 578-586.
40. McConkie, G. W., Zola, D., Grimes, J., Kerr, P. W., Bryant, N. R., & Wolff, P. M. (1991). Children's eye movements during reading. In J. F. Stein (Ed.), Vision and visual dyslexia 13. London: MacMillan.
41. McDonald, S. A., Carpenter, R. H. S., & Shillcock, R. C. (2005). An anatomically constrained, stochastic model of eye movement control in reading. Psychological Review, 112, 814-840.
42. Mouchetant-Rostaing, Y., Giard, M.-H., Bentin, S., Aguera, P.-E., & Pernier, J. (2000). Neurophysiological correlates of face gender processing in humans. European Journal of Neuroscience, 12, 303-310.
43. Nuthmann, A., & Engbert, R. (2009). Mindless reading revisited: An analysis based on the SWIFT model of eye-movement control. Vision Research, 49, 322-336.
44. Nuthmann, A., Engbert, R., & Kliegl, R. (2005). Mislocated fixations during reading and the inverted optimal viewing position effect. Vision Research, 45, 2201-2217.
45. O'Regan, J. K. (1992). Optimal viewing position in words and the strategy-tactics theory of eye movements in reading. In K. Rayner (Ed.), Eye movements and visual cognition: Scene perception and reading (pp. 333-354). New York: Springer-Verlag.
46. O'Regan, J. K., & Lévy-Schoen, A. (1987). Eye-movement strategy and tactics in word recognition and reading. In M. Coltheart (Ed.), Attention and performance, XII: The psychology of reading (pp. 329-348). Hillsdale, NJ: Erlbaum.
47. Pollatsek, A., Bolozky, S., Well, A. D., & Rayner, K. (1981). Asymmetries in the perceptual span for Israeli readers. Brain and Language, 14, 174-180.
48. Pollatsek, A., & Rayner, K. (1999). Is covert attention really unnecessary? Behavioral and Brain Sciences, 22, 695-696.
49. Pollatsek, A., Rayner, K., & Balota, D. A. (1986). Inferences about eye movement control from the perceptual span in reading. Perception & Psychophysics, 40, 123-130.
50. Pollatsek, A., Reichle, E. D., & Rayner, K. (2006a). Attention to one word at a time is still a viable hypothesis: Rejoinder to Inhoff, Eiter, and Radach. Journal of Experiment Psychology: Human Perception and Performance, 32, 1496-1500.
51. Pollatsek, A., Reichle, E. D., & Rayner, K. (2006b). Serial processing is consistent with the time course of linguistic information extraction from consecutive words during eye fixations in reading: A response to Inhoff, Eiter, and Radach (2005). Journal of Experiment Psychology: Human Perception and Performance, 32, 1485-1489.
52. Pynte, J., Kennedy, A., & Ducrot, S. (2004). The influence of parafoveal typographical errors on eye movements in reading. European Journal of Cognitive Psychology, 16, 178-202.
53. Radach, R., & Heller, D. (2000). Relations between spatial and temporal aspects of eye movements control. In A. Kennedy, R. Radach, D. Heller, & J. Pynte (Eds.), Reading as a perceptual process (pp. 165-192). Amsterdam: Elsevier.
54. Rayner, K. (1979). Eye movements and cognitive psychology: On-line computer approaches to studying visual information processing. Behavior Research Methods & Instruments, 11, 164-171.
55. Rayner, K. (1986). Eye movements and the perceptual span in beginning and skilled readers. Journal of Experimental Child Psychology, 41, 211-236.
56. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.
57. Rayner, K. (2009). The thirty-fifth Sir Frederick Bartlett lecture: Eye movements and attention during reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62, 1457-1506.
58. Rayner, K., Ashby, J., Pollatsek, A., & Reichle, E. D. (2004). The effects of frequency and predictability on eye fixations in reading: Implications for the E-Z Reader model. Journal of Experimental Psychology: Human Perception and Performance, 30, 720-732.
59. Rayner, K., & Bertera, J. H. (1979). Reading without a fovea. Science, 206, 468-469.
60. Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14, 191-201.
61. Rayner, K., & Fischer, M. H. (1996). Mindless reading revisited: Eye movements during reading and scanning are different. Perception & Psychophysics, 58, 734-747.
62. Rayner, K., & Juhasz, B. J. (2004). Eye movements in reading: Old questions and new directions. European Journal of Cognitive Psychology, 16, 340-352.
63. Rayner, K., & Morrison, R. M. (1981). Eye movements and identifying words in parafoveal vision. Bulletin of the Psychonomic Society, 17, 135-138.
64. Rayner, K., & Pollatsek, A. (1981). Eye movement control during rading: Evidence for direct control. Quarterly Journal of Experimental Psychology, 33A, 351-373.
65. Rayner, K., & Pollatsek, A. (1989). The psychology of reading. Englewood Cliffs, NJ: Erlbaum.
66. Rayner, K., Pollatsek, A., Drieghe, D., Slattery, T. J., & Reichle, E. D. (2007). Tracking the mind during reading via eye movements: Comments on Kliegl, Nuthmann, and Engbert (2006). Journal of Experimental Psychology: General, 136, 520-529.
67. Rayner, K., Pollatsek, A., Liversedge, S. P., & Reichle, E. D. (2009). Eye movements and non-canonical reading: Comments on Kennedy and Pynte (2008). Vision Research, 49, 2232-2236.
68. Rayner, K., Sereno, S. C., & Raney, G. E. (1996). Eye movement control in reading: A comparison of two types of models. Journal of Experimental Psychology: Human Perception and Performance, 22, 1188-1200.
69. Rayner, K., Slowiaczek, M. L., Clifton, C., & Bertera, J. H. (1983). Latency of sequential eye movements: Implications for reading. Journal of Experimental Psychology: Human Perception and Performance, 9, 912-922.
70. Rayner, K., & Well, A. D. (1996). Effects of contextual constraint on eye movements in reading: A further examination. Psychonomic Bulletin & Review, 3, 504-509.
71. Rayner, K., White, S. J., Kambe, G., Miller, B., & Liversedge, S. P. (2003). On the processing of meaning from parafoveal vision during eye fixations in reading. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind's eye: Cognitive and applied aspects of eye movements (pp. 213-234). Oxford, England: Oxford University Press.
72. Reichle, E. D. (2011). Serial attention models of reading. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), Oxford handbook on eye movements (pp. 767-786). Oxford, England: Oxford University Press.
73. Reichle, E. D., & Drieghe, D. (in press). Using E-Z Reader to examine word skipping during reading. Journal of Experimental Psychology: Learning Memory and Cognition.
74. Reichle, E. D., & Laurent, P. A. (2006). Using reinforcement learning to understand the emergence of "intelligent" eye-movement behavior during reading. Psychological Review, 113, 390-408.
75. Reichle, E. D., Liu, Y., & Laurent, P. A. (2011a). The emergence of adaptive eye movement control in reading: Theory and data. Studies of Psychology and Behavior, 9, 45-52.
76. Reichle, E. D., Liversedge, S. P., Pollatsek, A., & Rayner, K. (2009). Encoding multiple words simultaneously in reading is implausible. Trends in Cognitive Sciences, 13, 115-119.
77. Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105, 125-157.
78. Reichle, E. D., Pollatsek, A., & Rayner, K. (2012b). Using E-Z Reader to simulate eye movements in non-reading tasks: A unified framework for understanding the eye-mind link. Psychological Review, 119, 155-185.
79. Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The E-Z Reader model of eye movement control in reading: Comparisons to other models. Behavioral and Brain Sciences, 26, 445-476.
80. Reichle, E. D., Tokowicz, N., Liu, Y., & Perfetti, C. A. (2011b). Testing an assumption of the E-Z Reader model of eye-movement control during reading: Using event-related potentials to examine the familiarity check. Psychophysiology, 48, 993-1003.
81. Reilly, R., & O'Regan, K. J. (1998). Eye movement control during reading: A simulation of some word-targeting strategies. Vision Research, 38, 303-317.
82. Reilly, R. G., & Radach, R. (2006). Some empirical tests of an interactive activation model of eye movement control in reading. Journal of Cognitive Systems Research, 7, 34-55.
83. Reingold, E. M., Reichle, E. D., Glaholt, M. G., & Sheridan, H. (2012). Direct lexical control of eye movements in reading: Evidence from survival analysis of fixation durations. Cognitive Psychology, 65, 177-206.
84. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by backpropagating errors. Nature, 323, 533-536.
85. Salvucci, D. D. (2001). An integrated model of eye movements and visual encoding. Cognitive Systems Research, 1, 201-220.
86. Schad, D. J., & Engbert, R. (2012). The zoom lens of attention: Simulating shuffled versus normal text reading using the SWIFT model. Visual Cognition, 20, 391-421.
87. Schilling, H. E. H., Rayner, K., & Chumbley, J. I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory & Cognition, 26, 1270-1281.
88. Sereno, S. C., Rayner, K., & Posner, M. I. (1998). Establishing a time-line of word recognition: Evidence from eye movements and event-related potentials. NeuroReport, 9, 2195-2200.
89. Suppes, P. (1994). Stochastic models of reading. In J. Ygge & G. Lennerstrand (Eds.), Eye movements in reading (pp. 349-364). Oxford, England: Pergamon Press.
90. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
91. Suttorp, T., Hansen, N., & Igel, C. (2009). Efficient covariance matrix update for variable metric evolution strategies. Machine Learning, 75, 167-197.
92. Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136.
93. Van Rullen, R., & Thorpe, S. (2001). The time course of visual processing: From early perception to decision-making. Journal of Cognitive Neuroscience, 13, 454-461.
94. Vitu, F., McConkie, G. W., Kerr, P., & O'Regan, J. K. (2001). Fixation location effects on fixation durations during reading: An inverted optimal viewing position effect. Vision Research, 41, 3513-3533.
95. Vitu, F., O'Regan, J. K., Inhoff, A. W., & Topolski, R. (1995). Mindless reading: Eye-movement characteristics are similar in scanning letter strings and reading text. Perception & Psychophysics, 57, 352-364.
96. Yan, M., Kliegl, R., Richter, E. M., Nuthmann, A., & Shu, H. (2010). Flexible saccade-target selection in Chinese reading. Quarterly Journal of Experimental Psychology, 63, 705-725.
97. Yang, S.-N. (2006). A oculomotor-based model of eye movements in reading: The competition/activation model. Cognitive Systems Research, 7, 56-69.
98. Yang, S.-N., & McConkie, G. W. (2001). Eye movements during reading: A theory of saccade initiation time. Vision Research, 41, 3567-3585.