A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task

Neuroscience

Volumn 91, Issue 3, 1999, Pages 871-890

  Suri, R E ^a   Schultz, W ^a  

^a UNIVERSITY OF FRIBOURG   (Switzerland)

Author keywords

Basal ganglia; Striatum; Synaptic plasticity; Teaching signal; Temporal difference

Indexed keywords

ALGORITHM; ANIMAL EXPERIMENT; ARTICLE; BASAL GANGLION; BODY MOVEMENT; COGNITION; CONDITIONED REFLEX; DOPAMINERGIC NERVE CELL; EXPECTATION; FRONTAL CORTEX; MONKEY; NERVE CELL NETWORK; NONHUMAN; PARKINSON DISEASE; PRIORITY JOURNAL; REINFORCEMENT; SHORT TERM MEMORY; SPATIAL DISCRIMINATION; STIMULUS RESPONSE; TASK PERFORMANCE;

ANIMALS; BEHAVIOR, ANIMAL; CHOICE BEHAVIOR; CONDITIONING (PSYCHOLOGY); DOPAMINE; LEARNING; MACACA; MODELS, NEUROLOGICAL; NERVE NET; REACTION TIME; REINFORCEMENT (PSYCHOLOGY); REWARD; SPATIAL BEHAVIOR;

EID: 0032930935     PISSN: 03064522     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0306-4522(98)00697-6     Document Type: Article

References (86)

1. Alexander G.E Selective neuronal discharge in monkey putamen reflects intended direction of planned limb movements. Expl Brain Res. 67:1987;623-634.
2. Apicella P., Scarnati E., Ljungberg T., Schultz W. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68:1992;945-960.
3. Barto A. G. (1995) Adaptive Critics and the Basal Ganglia. In Models of Information Processing in the Basal Ganglia (eds Houk J. C., Davis J. L. and Beiser D. G.), pp. 215-232. MIT, Cambridge.
4. Barto A. G., Sutton R. S. and Anderson C. W. (1983) Neuronlike adaptive elements that can solve difficult learning control ploblems. IEEE Trans on Systems, Man, and Cybernetics SMC-13, pp. 834-846.
5. Barto A. G., Sutton R. S. and Watkins C. J. C. H. (1990) Learning and sequential decision making. In Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel M. and Moore J.), pp. 539-602. MIT, Cambridge.
6. Bauer R.H, Fuster J.M Delayed-matching and delayed-response deficit from cooling dorsolateral prefrontal cortex in monkeys. J. comp. Physiol. Psychol. 90:1976;293-302.
7. Beiser D.G, Houk J.C Model of cortical-basal ganglionic processing: encoding the serial order of sensory events. J. Neurophysiol. 79(6):1998;3168-3188.
8. Bishop G.A, Chang H.T, Kitai S.T Morphological and physiological properties of neostriatal neurons: an intracellular horseradish peroxidase study in the rat. Neuroscience. 7:1982;179-191.
9. Blackburn J.R, Pfaus J.G, Phillips A.G Dopamine functions in appetitive and defensive behaviours. Prog. Neurobiol. 39(3):1992;247-279.
10. Bouyer J.J, Park D.H, Joh T.H, Pickel V.M Chemical and structural analysis of the relation between cortical inputs and tyrosine hydroxylase-containing terminals in rat neostriatum. Brain Res. 302:1984;267-275.
11. Calabresi P., Pisani A., Mercuri N.B, Bernardi G. Long-term potentiation in the striatum is unmasked by removing the voltage-dependent magnesium block of NMDA receptor channels. Eur. J. Neurosci. 4:1992;929-935.
12. Calabresi P., Saiardi A., Pisani A., Baik J.H, Centonze D., Mercuri N.B, Bernardi G., Borrelli E. Abnormal synaptic plasticity in the striatum of mice lacking dopamine D2 receptors. J. Neurosci. 17(12):1997;4536-4544.
13. Church R. M. (1978) The internal clock. In Cognitive Processes in Animal Behavior (eds Hulse S. H., Fowler H. and Honig W. K), Lawrence Erlbaum Associates, Hillsdale, New Jersey.
14. Contreras-Vidal, J. L. and Schultz, W. (1999) A predictive reinforcement model of dopamine neurons for learning approach behavior. J. comput. Neurosci. (in press).
15. Dickinson A. (1980) Contemporary Animal Learning Theory. Cambridge University Press, Cambridge.
16. Fagg, A. H. (1993) Reinforcement learning for robotic reaching and grasping. In New Perspectives in the Control of the Reach to Grasp Movement (eds Bennet K. M. B. and Castiello U.), pp. 281-308. North Holland.
17. Freund T.F, Powell J.F, Smith A.D Tyrosine hydroxylase-immunoreactive boutons in synaptic contact with identified striatonigral neurons, with particular reference to dendritic spines. Neuroscience. 13:1984;1189-1215.
18. Friston K.J, Tononi G., Reeke G.N Jr., Sporns O., Edelman G.M Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience. 59:1994;229-243.
19. Funahashi S., Bruce C.J, Goldman-Rakic P.S Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J. Neurophysiol. 61:1989;331-349.
20. Fuster J.M, Alexander G.E Neuron activity related to short-term memory. Science. 173:1971;652-654.
21. Gallistel C. R. (1990) The Organization of Learning. A Bradford Book, MIT, Cambridge, Massachusetts.
22. Graybiel A.M, Aosaki T., Flaherty A.W, Kimura M. The basal ganglia and adaptive motor control. Science. 265:1994;1826-1831.
23. Grossberg S. Adaptive pattern classification and universal recording: I. Parallel development and coding of neural feature detectors. Biol. Cybern. 23:1976;121-134.
24. Grossberg S., Levine D.S Neural dynamics of attentionally modulated Pavlovian conditioning: blocking, interstimulus interval, and secondary reinforcement. Appl. Optics. 26(23):1987;5015-5030.
25. Grossberg S., Schmajuk N.A Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Networks. 2:1989;79-102.
26. Groves P.M, Linder J.C, Young S.J 5-hydroxydopamine-labeled dopamine axons: three-dimensional reconstructions of axons, synapses and postsynaptic targets in rat neostriatum. Neuroscience. 58:1994;593-604.
27. Guigon E., Dorizzi B., Burnod Y., Schultz W. Neural correlates of learning in the prefrontal cortex of the monkey: a predictive network model. Cerebr. Cortex. 5:1995;135-147.
28. Gullapalli V., Barto A. G. and Grupen R. A. (1994) Learning admittance mapping for force-guided assembly. In Proceedings of the 1994 International Conference on Robotics and Automation, pp. 2633-2638. Computer Society Press, Los Alamitos, California.
29. Hollerman J., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1:1998;304-309.
30. Houk J. C., Adams J. L. and Barto A. G. (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In Models of Information Processing in the Basal Ganglia (eds Houk J. C., Davis J. L. and Beiser D. G.), pp. 215-232. MIT, Cambridge, Massachusetts.
31. Hrupka B.J, Lin Y.M, Gietzen D.W, Rogers Q.R Small changes in essential amino acid concentrations alter diet selection in amino acid-deficient rats. J. Nutr. 127(5):1997;777-784.
32. Hull C. L. (1943) Principles of Behavior. Appleton-Century-Crofts, New York.
33. Jacobsen C.F, Nissen H.W Studies of cerebral function in primates: IV. The effects of frontal lobe lesions on the delayed alternation habit in monkeys. J. comp. Physiol. Psychol. 23:1937;101-112.
34. Jaeger D., Hitoshi K., Wilson C.J Surrond inhibition among projection neurons is weak of nonexistent in the rat neostriatum. J. Neurophysiol. 72(3):1994;2555-2558.
35. Kalman R.E A new aproach to linear filtering and prediction problems. J. Basic Eng. Trans. ASME. 82:1960;35-45.
36. Kawaguchi Y., Wilson C.J, Emson P.C Projection subtypes of rat neostriatal matrix cells revealed by intracellular injection of biocytin. J. Neurosci. 10:1990;3421-3438.
37. Klopf A.H A neuronal model of classical conditioning. Psychobiology. 16(2):1988;85-125.
38. Knowlton B.J, Mangels J.A, Squire L.R A neostriatal habit learning system in humans. Science. 273(5280):1996;1399-1402.
39. Kubota K., Niki H. Prefrontal cortical unit activity and delayed alternation performance in monkeys. J. Neurophysiol. 34:1971;337-347.
40. Lees A.J, Smith E. Cognitive deficits in the early stages of Parkinson's disease. Brain. 106:1983;257-270.
41. Linden A., Bracke-Tolkmitt R., Lutzenberger W., Canavan A.G.M., Scholz E., Diener H.C, Birbaumer N. Slow cortical potentials in Parkinsonian patients during the course of an associative learning task. J. Psychophysiol. 4:1990;145-162.
42. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67:1992;145-163.
43. Mackintosh N. M. (1983) Theoretical analysis of instrumental conditioning. In Conditioning and Associative Learning (ed. Mackintosh N. M.), pp. 77-112. Oxford University Press, Oxford.
44. Mark G.P, Blander D.S, Hoebel B.G A conditioned stimulus decreases extracellular dopamine in the nucleus accumbens after the development of a learned taste aversion. Brain Res. 551:1991;308-310.
45. Mason S.T, Beninger R.J, Fibiger H.C, Phillips A.G Pimozide-induced suppression of responding: evidence against a block of food reward. Pharmac. Biochem. Behav. 12:1980;917-923.
46. Mirenowicz J., Schultz W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72:1994;1024-1027.
47. Mirenowicz J., Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature. 379:1996;449-451.
48. Montague P. R., Dayan P., Nowlan S. J., Pouget A. and Sejnowski T. J. (1993) Using aperiodic reinforcement for directed self-organization. In Advances in Neural Information Processing Systems (eds Giles C. L., Hanson S. J. and Cowan J. D.), pp.969-976. Morgan Kaufmann, San Mateo, California.
49. Montague P.R, Dayan P., Person C., Sejnowski T.J Bee foraging in uncertain environments using predictive hebbian learning. Nature. 377:1995;725-728.
50. Montague P.R, Dayan P., Sejnowski T.J A framework for mesencephalic dopamine systems based on predictive hebbian learning. J. Neurosci. 16(5):1996;1936-1947.
51. Moody S.L, Wise S.P, di Pellegrino G., Zipser D. A model that accounts for activity in primate frontal cortex during a delayed matching-to-sample task. J. Neurosci. 18:1998;399-410.
52. Muir J.L, Everitt B.J, Robbins T.W The cerebellar cortex of the rat and visual attentional function: dissociate effects of medio-frontal, cingulate, anterior dorsolateral, and parietal cortex lesion on a five choice serial reaction time task. Cerebr. Cortex. 6:1996;470-481.
53. 2+ responses. Nature. 354:1991;73-76.
54. Percheron G., Francois C., Yelnick J. and Fenelon G. (1989) The primate nigro-striato-pallido-nigral system. Not a mere loop. In Neural Mechanisms in Disorders of Movement (eds Crossman A. R. and Sambrook M. A.), pp. 103-109. John Libbey, London.
55. Rasmussen K., Strecker R.E, Jacobs B. Single unit response of noradrenegic, serotonergic and dopamine neurons in freely moving cats to simple sensory stimuli. Brain Res. 369:1986;336-340.
56. Rescorla R. A. and Wagner A. R. (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. In Classical Conditioning II: Current Research and Theory (eds Black A. H. and Prokasy W. F.), Appleton-Century-Crofts, New York.
57. Revusky S. (1971) The role of interference in association over a delay. In Animal Memory (eds Honig W. K. and James P. H. R.), pp. 155-213. Academic, New York.
58. Robinson T.E, Berridge K.C The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18:1993;247-291.
59. Rogers Q.R, Harper A.E Selection of a solution containing histidine by rats fed a histidine-imbalanced diet. J. comp. Physiol. Psychol. 72:1970;66-71.
60. Romo R., Schultz W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J. Neurophysiol. 63:1990;592-606.
61. Salamone J.D, Cousins M.S, Snyder B.J Behavioral functions of nucleus accumbens dopamine: empirical and conceptual problems with the anhedonia hypothesis. Neurosci. biobehav. Rev. 21(3):1997;341-359.
62. Schultz W. Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J. Neurophysiol. 56:1986;1439-1462.
63. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80:1998;1-27.
64. Schultz W., Apicella P., Romo R. and Scarnati E. (1995) Context-dependent activity in primate striatum reflecting past and future events. In Models of Information Processing in the Basal Ganglia (eds Houk J. C., Davis J. L. and Beiser D. G.), pp. 11-27. MIT, Cambridge.
65. Schultz W., Apicella P., Scarnati E., Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci. 12:1992;4595-4610.
66. Schultz W., Apicella P., Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13(3):1993;900-913.
67. Schultz W., Dayan P., Montague P.R A neural substrate of prediction and reward. Science. 275(5306):1997;1593-1599.
68. Schultz W., Romo R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57(1):1987;201-217.
69. Schultz W., Romo R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J. Neurophysiol. 63:1990;607-624.
70. Smith A.D, Bolam J.P The neural network of the basal ganglia as revealed by the study of synaptic connections of identified neurons. Trends Neurosci. 13:1990;259-265.
71. Sprengelmeyer R., Canavan A.G.M., Lange H.W, Hömberg V. Associative learning in degenerative neostriatal disorders: contrasts in explicit and implicit remembering between Parkinson's and Huntington's disease patients. Mov. Disord. 10:1995;85-91.
72. Steinfels G.F, Heym J., Strecker R.E, Jacobs B.L Behavioral correlates of dopamine unit activity in freely moving cats. Brain Res. 258:1983;217-228.
73. Strecker R.E, Jacobs B.L Substantia nigra dopamine unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res. 361:1985;339-350.
74. Suri R., Schultz W. A neural learning model based on the activity of primate dopamine neurons. Soc. Neurosci. Abstr. 22:1996;1389.
75. Suri R.E, Schultz W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Expl Brain Res. 121:1998;350-354.
76. Sutton R. S. (1984) Temporal Credit Assignment in Reinforcement Learning. PhD thesis, Department of Computer and Information Science, University of Massachusetts, Amherst, MA, USA.
77. Sutton R.S, Barto A.G Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88:1981;135-170.
78. Sutton R. S. and Barto A. G. (1990) Time derivative models of Pavlovian reinforcement. In Learning and Computational Neuroscience: Foundations of Adaptive Networks (eds Gabriel M. and Moore J.), pp. 539-602. MIT, Cambridge.
79. Sutton R. S. and Barto A. G. (1998) Section 2.7 Optimistic Initial Values. In Reinforcement Learning: An Introduction. pp. 39-41. MIT Press/Bradford Books, Cambridge, MA.
80. Tanji J., Taniguchi K., Saga T. Supplementary motor area: neuronal responses to motor instructions. J. Neurophysiol. 43:1980;60-68.
81. Tesauro G. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6:1994;215-219.
82. Vriezen E.R, Moscovitch M. Memory for temporal order and conditional associative-learning in patients with Parkinson's disease. Neuropsychologia. 28:1990;1283-1293.
83. Weinrich M., Wise S.P The premotor cortex of the monkey. J. Neurosci. 2:1982;1329-1345.
84. Wickens J.R, Begg A.J, Arbuthnott G.W Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience. 70(3):1996;1-5.
85. Wickens J. and Kötter R. (1995) Cellular models of reinforcement. In Models of Information Processing in the Basal Ganglia, (eds Houk J. C., Davis J. L. and Beiser D. G.), pp. 187-214. MIT, Cambridge, Massachusetts.
86. Wise R.A, deWit J.S.H., Gerber G.J Neuroleptic-induced 'anhedonia' in rats: pimozide blocks reward quality of food. Science. 201:1978;262-264.