Attention-gated reinforcement learning of internal representations for classification

Neural Computation

Volumn 17, Issue 10, 2005, Pages 2176-2214

  Roelfsema, Pieter R ^a,b   Van Ooyen, Arjen ^a,b  

^a NETHERLANDS INSTITUTE FOR NEUROSCIENCE   (Netherlands)

^b VU UNIVERSITY AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; ARTICLE; ARTIFICIAL NEURAL NETWORK; ATTENTION; BIOLOGICAL MODEL; CLASSIFICATION; COMPARATIVE STUDY; FEEDBACK SYSTEM; HUMAN; LEARNING; NERVE CELL; NERVE CELL PLASTICITY; PHYSIOLOGY; REINFORCEMENT;

ANIMALS; ATTENTION; FEEDBACK; GENERALIZATION (PSYCHOLOGY); HUMANS; LEARNING; MODELS, NEUROLOGICAL; NEURAL NETWORKS (COMPUTER); NEURONAL PLASTICITY; NEURONS; REINFORCEMENT (PSYCHOLOGY);

EID: 23944437453     PISSN: 08997667     EISSN: None     Source Type: Journal    
DOI: 10.1162/0899766054615699     Document Type: Article

References (96)

1. Ahissar, M., & Hochstein, S. (1993). Attentional control of early perceptual learning. Proc. Natl. Acad. Sci. USA, 90, 5718-5722.
2. Baker, C. I., Behrmann, M., & Olson, C. R. (2002). Impact of learning on representation of parts and wholes in monkey inferotemporal cortex. Nat. Neurosci., 5, 1210-1216.
3. Bakin, J. S., & Weinberger, N. M. (1996). Induction of a physiological memory in the cerebral cortex by stimulation of the nucleus basalis. Proc. Natl. Acad. Sci. USA, 93, 11219-11224.
4. Bao, S., Chan, V. T., & Merzenich, M. M. (2001). Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature, 412, 79-81.
5. Barto, A. G. (1985). Learning by statistical cooperation of self-interested neuron-like computing elements. Human Neurobiol., 4, 229-256.
6. Barto, A. G., & Anandan, P. (1985). Pattern-recognizing stochastic learning automata. IEEE Trans. Syst., Man. and Cybernet., SMC-15, 360-375.
7. Baxter, J., & Bartlett, P. L. (2001). Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15, 319-350.
8. Bear, M. F., & Singer, W. (1986). Modulation of visual cortical plasticity by acetylcholine and noradrenaline. Nature, 320, 172-176
9. Becker, S., & Hinton, G. E. (1992). Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355, 161-163.
10. Bishop, C. M. (1995). Neural networks for pattern recognition. Oxford: Oxford University Press.
11. Boch, R., & Fischer, B. (1983). Saccadic reaction times and activation of the prelunate cortex: Parallel observations in trained rhesus monkeys. Exp. Brain Res., 50, 201-210.
12. Chelazzi, L., Miller, E. K., Duncan, J., & Desimone, R. (1993). A neural basis for visual search in inferior temporal cortex. Nature, 363, 345-347.
13. Colby, C. L., Duhamel, J. R., & Goldberg, M. E. (1996). Visual, presaccadic, and cognitive activation of single neurons in monkey lateral intraparietal area. J. Neurophysiol, 76, 2841-2852.
14. Collingridge, G. L., & Bliss, P. (1987). NMDA receptors - their role in long-term potentiation. Trends Neurosci., 10, 288-293.
15. Crick, F. (1989). The recent excitement about neural networks. Nature, 337, 129-132.
16. Dayan, P., & Balleine, B. W. (2002). Reward, motivation, and reinforcement learning. Neuron, 36, 285-298.
17. Dehaene, S., Sergent, C., & Changeux, J. P. (2003). A neuronal network model linking subjective reports and objective physiological data during conscious perception. Proc. Natl. Acad. Sci. USA, 100, 8520-8525.
18. Desimone, R., & Duncan, J. (1995). Neural mechanisms of selective visual attention. Annu. Rev. Neurosci., 18, 193-222.
19. Deubel, H., & Schneider, W. X. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision Res., 36, 1827-1837.
20. Douglas, R. J., Koch, C., Mahowald, M., Martin, K. A. C., & Suarez, H. H. (1995). Recurrent excitation in neocortical circuits. Science, 269, 981-985.
21. Easton, A., Ridley, R. M., Baker, H. F., & Gaffan, D. (2002). Unilateral lesions of the cholinergic basal forebrain and fornix in one hemisphere and inferior temporal cortex in the opposite hemisphere produce severe learning impairements in rhesus monkeys. Cereb. Cortex, 12, 729-736.
22. Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex, 1, 1-47.
23. Fiorillo, C. D., Tobler, P. N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898-1902.
24. Fox, K., Sato, H., & Daw, N. (1990). The effect of varying stimulus intensity on NMDA-receptor activity in cat visual cortex. J. Neurophysiol., 64, 1413-1428.
25. Freedman, D. J., Riesenhuber, M., Poggio, T., & Miller, E. K. (2001). Categorical representation of visual stimuli in the primate prefrontal cortex. Science, 291, 312-316.
26. Freedman, D. J., Riesenhuber, M., Poggio, T., & Miller, E. K. (2003). A comparison of primate prefrontal cortex and inferior temporal cortices during visual categorization. J. Neurosci., 15, 5235-5246.
27. Garris, P. A., Kilpatrick, M., Bunin, M. A., Michale, D., Walker, Q. D., & Wightman, R. M. (1999). Dissociation of dopamine release in the nucleus accumbens from intracranial self-stimulation. Nature, 398, 67-69.
28. Gold, J. I., & Shadlen, M. N. (2001). Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci., 5, 10-16.
29. Goldberg, M. E., & Segraves, M. A. (1987). Visuospatial and motor attention in the monkey. Neuropsychologia, 25, 107-118.
30. Gorman, R. P., & Sejnowski, T. (1988). Analysis of hidden units in a layered network trained to classify sonar targets. Neural Networks, 1, 75-89.
31. Gottlieb, J. P., Kusunoki, M., & Goldberg, M. E. (1998). The representation of visual salience in monkey parietal cortex. Nature, 391, 481-484.
32. Gurden, H., Takita, M., & Jay, T. M. (2000). Essential role of D1 but not D2 receptors in the NMDA receptor-dependent long-term potentiation at hippocampalprefrontal cortex synapses in vivo. J. Neurosci., 20, RC106 (1-5).
33. Gustafsson, B., & Wigstrom, H. (1988). Physiological mechanisms underlying long-term potentiation. Trends Neurosci., 11, 156-162.
34. Hinton, G. E., Dayan, P., Frey, B. J., & Neal, R. M. (1995). The "wake-sleep" algorithm for unsupervised neural networks. Science, 268, 1158-1161.
35. Hoffman, J. E., & Subramaniam, B. (1995). The role of visual attention in saccadic eye movements. Percept. Psychophys., 57, 787-795.
36. Juliano, S. L., Ma, W., & Eslin, D. (1991). Cholinergic depletion prevents expansion of topographic maps in somatosensory cortex. Proc. Natl. Acad. Sci. USA, 88, 780-784.
37. Kilgard, M. P., & Merzenich, M. M. (1998). Cortical map reorganization enabled by nucleus basalis activity. Science, 279, 1714-1718.
38. Körding, K. P., & König, P. (2001). Supervised and unsupervised learning with two sites of synaptic integration. J. Comp. Neurosci., 11, 207-215.
39. Kowler, E., Anderson, E., Dosher, B., & Blaser, E. (1995). The role of attention in the programming of saccades. Vision Res., 35, 1897-1916.
40. Lamme, V. A. F., & Roelfsema, P. R. (2000). The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci., 23, 571-579.
41. Ljungberg, T., Apicella, P., & Schultz, W. (1993). Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol., 67, 145-163.
42. Luck, S. J., Chelazzi, L., Hillyard, S. A., & Desimone, R. (1997). Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex. J. Neurophysiol., 77, 24-42.
43. Malinow, R., & Miller, J. P. (1986). Postsynaptic hyperpolarisation during conditioning reversibly blocks induction of long-term potentiation. Nature, 320, 529-530.
44. Mazzoni, P., Andersen, R. A., & Jordan, M. I. (1991). A more biologically plausible learning rule for neural networks. Proc. Natl. Acad. Sci. USA, 88, 4433-4437.
45. McAdams, C. J., & Maunsell, J. H. R. (1999). Effects of attention on orientation-tuning functions of single neurons in macaque area V4. J. Neurosci., 19, 431-441.
46. Montague, P. R., & Berns, G. S. (2002). Neural economics and the biological substrates of valuation. Neuron, 36, 265-284.
47. Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728.
48. Moore, T. (1999). Shape representations and visual guidance of saccadic eye movements. Science, 285, 1914-1917.
49. Moore, T., & Armstrong, K. M. (2003). Selective gating of visual signals by microstimulation of frontal cortex. Nature, 421, 370-373.
50. Moran, J., & Desimone, R. (1985). Selective attention gates visual processing in the extrastriate cortex. Science, 229, 782-784.
51. Morris, G., Arkadir, D., Nevet, A., Vaadia, E., & Bergman, H. (2004). Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron, 43, 133-143.
52. Motter, B. C. (1993). Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli. J. Neurophysiol., 70, 909-919.
53. Muller, D., Joly, M., & Lynch, G. (1988). Contributions of quisqualate and NMDA receptors to the induction and expression of LTP. Science, 242, 1694-1697.
54. Nowlan, S. J., & Sejnowski, T. J. (1995). A selection model for motion processing in area MT of primates. J. Neurosci., 15, 1195-1214.
55. O'Reilly, R. C. (1996). Biologically plausible error-driven learning using local activation differences: The generalized recirculation algorithm. Neural Comp., 8, 895-938.
56. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
57. Otani, S., Auclair, N., Desce, J. M., Roisin, M. P., & Crepel, F. (1999). Dopamine receptors and groups I and II mGluRs cooperate for long-term depression induction in rat prefrontal cortex through converging postsynaptic activation of MAP kinases. J. Neurosci., 15, 9788-9802.
58. Pennartz, C. A. M. (1995). The ascending neuromodulatory systems in learning by reinforcement: Comparing computational conjectures with experimental findings. Brain Res. Rev., 21, 219-245.
59. Phillips, P. E. M., Stuber, G. D., Heien, M. L. H. V., Wightman, R. M., & Carelli, R. M. (2003). Subsecond dopamine release promotes cocaine seeking. Nature, 422, 614-618.
60. Reynolds, J. H., Pasternak, T., & Desimone, R. (2000). Attention increases sensitivity of V4 neurons. Neuron, 26, 703-714.
61. Reynolds, J. N. J., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413, 67-70.
62. Rizzolatti, G., Riggio, L., & Sheliga, B. M. (1994). Space and selective attention. In C. Umiltà & M. Moscovitch (Eds.), Attention and performance XV. Conscious and nonconscious information processing (pp. 231-265). Cambridge, MA: MIT Press.
63. Roelfsema, P. R., Lamme, V. A. F., & Spekreijse, H. (1998). Object-based attention in the primary visual cortex of the macaque monkey. Nature, 395, 376-381.
64. Roelfsema, P. R., Lamme, V. A. F., & Spekreijse, H. (2004). Synchrony and covariation of firing rates in the primary visual cortex during contour grouping. Nature Neurosci., 7, 982-991.
65. Rosenkranz, J. A., & Grace, A. A. (2002). Dopamine-mediated modulation of odour-evoked amygdala potentials during Pavlovian conditioning. Nature, 417, 282-287.
66. Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal representations by error propagation. In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group (Eds.), Parallel distributed processing (Vol. 1, pp. 318-364). Cambridge, MA: MIT Press.
67. Rumelhart, D. E., & Zipser, D. (1986). Feature discovery by competitive learning. In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group (Eds.), Parallel distributed processing (Vol. 1, pp. 151-193). Cambridge, MA: MIT Press.
68. Salin, P. A., & Bullier, J. (1995). Corticocortical connections in the visual system: Structure and function. Physiol. Rev., 75, 107-154.
69. Schall, J. D., & Hanes, D. P. (1993). Neural basis of saccade target selection in frontal eye field during visual search. Nature, 366, 467-469.
70. Schoups, A., Vogels, R., Qian, N., & Orban, G. A. (2001). Practising orientation identification improves orientation coding in V1 neurons. Nature, 412, 549-553.
71. Schultz, W. (2002). Getting formal with dopamine and reward. Neuron, 36, 241-263.
72. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
73. Schultz, W., & Dickinson, A. (2000). Neuronal coding of prediction errors. Annu. Rev. Neurosci., 23, 473-500.
74. Seidemann, E., Arieli, A., Grinvald, A., & Slovin, H. (2002). Dynamics of depolarization and hyperpolarization in the frontal cortex and saccade goal. Science, 295, 862-865.
75. Seung, H. S. (2003). Learning in spiking neural networks by reinforcement of stochastic synaptic transmission. Neuron, 40, 1063-1073.
76. Shi, S.-H., Hayashi, Y., Petralia, R. S., Zaman, S. H., Wenthold, R. J., Svoboda, K., & Malinow, R. (1999). Rapid spine delivery and redistribution of AMPA receptors after synaptic NMDA receptor activation. Science, 284, 1811-1816.
77. Shima K., & Tanji J. (1998). Involvement of NMDA and non-NMDA receptors in the neuronal responses of the primary motor cortex to input from the supplementary motor area and somatosensory cortex: Studies of task-performing monkeys. Jpn. J. Physiol., 48, 275-290.
78. Sigala, N., & Logothetis, N. K. (2002). Visual categorization shapes feature selectivity in the primate temporal cortex. Nature, 415, 318-320.
79. Sugase, Y., Yamane, S., Ueno, S., & Kawano, K. (1999). Global and fine information coded by single neurons in the temporal visual cortex. Nature, 400, 869-873.
80. Supèr, H., van der Togt, C., Spekreijse, H., & Lamme, V. A. F. (2004). Correspondence of presaccadic activity in the monkey primary visual cortex with saccadic eye movements. Proc. Natl. Acad. Sci. USA, 101, 3230-3235.
81. Suri, R. E., & Schultz, W. (2001). Temporal difference model reproduces anticipatory neural activity. Neural Comp., 13, 841-862.
82. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
83. Szabo M., Almeida R., Deco G., & Stetter, M. (in press). A neuronal model for the shaping of feature selectivity in IT by visual categorization. Neurocomputing.
84. Tanaka, K. (1995). Neuronal mechanisms of object recognition. Science, 262, 685-688.
85. Tesauro, G. (1995). Temporal difference learning and TD-Gammon. Comm. ACM, 38, 58-68.
86. Treue, S., & Martínez Trujillo, J. C. (1999). Feature-based attention influences motion processing gain in macaque visual cortex. Nature, 399, 575-579.
87. Treue, S., & Maunsell, J. H. R. (1996). Attentional modulation of visual motion processing in cortical areas MT and MST. Nature, 382, 539-541.
88. Usher, M., & McClelland, J. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychol. Rev., 108, 550-592.
89. van Ooyen, A., & Nienhuis, B. (1992). Improving the convergence of the backpropagation algorithm. Neural Networks, 5, 465-471.
90. Vogels, R., & Orban, G. A. (1990). How well do response changes in striate neurons signal differences in orientation? A study in the discriminating monkey. J. Neurosci., 10, 3543-3558.
91. Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43-48.
92. Warburton, E. C., Koder, T., Cho, K., Massey, P. V., Duguid, G., Barker, G. R. I., Aggleton, J. P., Bashir, Z. I., & Brown, M. W. (2003). Cholinergic neurotransmission is essential for perirhinal cortical plasticity and recognition memory. Neuron, 38, 987-996.
93. Werbos, P. J. (1974). Beyond regression: New tools for prediction and analysis in the behavioral sciences. Unpublished doctoral dissertation, Harvard University.
94. Winkler, J., Suhr, S. T., Gage, F. H., Thal, L. J., & Fisher, L. J. (1995). Essential role of neocortical acetylcholine in spatial memory. Nature, 375, 484-487.
95. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8, 229-256.
96. Zipser, D., & Rumelhart, D. E. (1990). The neurobiological significance of the new learning models. In E. L. Schwartz (Ed.), Computational neuroscience (pp. 192-200). Cambridge, MA: MIT Press.