Reward, motivation, and reinforcement learning

Neuron

Volumn 36, Issue 2, 2002, Pages 285-298

  Dayan, Peter ^a   Balleine, Bernard W ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

BEHAVIOR; CHEMICAL ANALYSIS; HABIT; LEARNING; MOTIVATION; NERVOUS SYSTEM; PRIORITY JOURNAL; REINFORCEMENT; REVIEW; REWARD;

EID: 0037057808     PISSN: 08966273     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0896-6273(02)00963-7     Document Type: Review

References (122)

1. Alexander G.E., Crutcher M.D. Functional architecture of basal ganglia circuits. neural substrates of parallel processing Trends Neurosci. 13:1990;266-271.
2. Baird L.C. Advantage Updating. Technical report WL-TR-93-1146. 1993;Wright-Patterson Air Force Base, Dayton, OH.
3. Balleine B.W. The role of incentive learning in instrumental performance following shifts in primary motivation. J. Exp. Psychol. Anim. Behav. Process. 18:1992;236-250.
4. Balleine B.W. Asymmetrical interactions between thirst and hunger in Pavlovian-instrumental transfer. Q. J. Exp. Psychol. B. 47:1994;211-231.
5. Balleine B.W. Incentive processes in instrumental conditioning. Mowrer R., Klein S. Handbook of Contemporary Learning Theories. 2001;Erlbaum, Hillsdale, New Jersey. 307-366.pp.
6. Balleine B.W., Dickinson A. Instrumental performance following reinforcer devaluation depends upon incentive learning. Q. J. Exp. Psychol. B. 43:1991;279-296.
7. Balleine B.W., Dickinson A. Signalling and incentive processes in instrumental reinforcer devaluation. Q. J. Exp. Psychol. B. 45:1992;285-301.
8. Balleine B.W., Dickinson A. Goal-directed instrumental action. contingency and incentive learning and their cortical substrates Neuropharmacology. 37:1998;407-419.
9. Balleine B.W., Dickinson A. Effect of lesions of the insular cortex on instrumental conditioning. evidence for a role in incentive memory J. Neurosci. 20:2000;8954-8964.
10. Balleine B.W., Killcross A.S. Effects of ibotenic acid lesions of the nucleus accumbens on instrumental action. Behav. Brain Res. 65:1994;181-193.
11. Balleine B.W., Ball J., Dickinson A. Benzodiazepine-induced outcome revaluation and the motivational control of instrumental action. Behav. Neurosci. 108:1994;573-589.
12. Barto A.G., Sutton R.S., Anderson C.W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics. 5:1983;834-846.
13. Berridge K.C. Reward learning. reinforcement, incentives, and expectations Medin D.L. The Psychology of Learning and Motivation. Advances in Research and Theory, Volume 40:2001;Academic Press, San Diego. 223-278. pp.
14. Berridge K.C., Robinson T.E. What is the role of dopamine in reward. hedonic impact, reward learning, or incentive salience? Brain Res. Brain Res. Rev. 28:1998;309-369.
15. Berridge K.C., Schulkin J. Palatability shift of a salt-associated incentive during sodium depletion. Q. J. Exp. Psychol. B. 41:1989;121-138.
16. Bertino M., Beauchamp G.K., Engelman K. Naltrexone, an opioid blocker, alters taste perception and nutrient intake in humans. Am. J. Physiol. 261:1991;R59-R63.
17. Bindra D. A motivational view of learning, performance, and behavior modification. Psychol. Rev. 81:1974;199-213.
18. Bindra D. How adaptive behavior is produced. a perceptual motivational alternative to response-reinforcement Behav. Brain Sci. 1:1978;41-52.
19. Bolles R.C. Theory of Motivation. 1975;Harper & Row, New York.
20. Braun J.J. Gustatory cortex. definition and function Kolb B., Tees R.C. The Cerebral Cortex of the Rat. 1990;MIT Press, Cambridge, MA. 407-430.pp.
21. Byrnes E.M., Reilly A., Bruno J.P. Effects of AMPA and D1 receptor activation on striatal and nigral GABA efflux. Synapse. 26:1997;254-268.
22. Cador M., Robbins T.W., Everitt B.J. Involvement of the amygdala in stimulus-reward associations. interaction with the ventral striatum Neuroscience. 30:1989;77-86.
23. Clarke S.N.D.A., Parker L.A. Morphine-induced modification of quinine palatability. effects of multiple morphine-quinine trials Pharmacol. Biochem. Behav. 51:1995;505-508.
24. Colwill R.C., Rescorla R.A. Associative structures in instrumental learning. Bower G.H. The Psychology of Learning and Motivation, Volume 20. 1986;Academic Press, New York. 55-104).pp.
25. Colwill R.M., Motzkin D.K. Encoding of the unconditioned stimulus in Pavlovian conditioning. Anim. Learn. Behav. 22:1994;384-394.
26. Colwill R.M., Rescorla R.A. Associations between the discriminative stimulus and the reinforcer in instrumental learning. J. Exp. Psychol. Anim. Behav. Process. 14:1988;155-164.
27. Corbit L., Muir J., Balleine B.W. The role of the nucleus accumbens in instrumental conditioning. evidence for a functional dissociation between accumbens core and shell J. Neurosci. 21:2001;3251-3260.
28. Davidson T.L., Altizer A.M., Benoit S.C., Walls E.K., Powley T.L. Encoding and selective activation of "metabolic memories" in the rat. Behav. Neurosci. 111:1997;1014-1030.
29. Davis J., Bitterman M.E. Differential reinforcement of other behavior (DRO). a yoked-control comparision J. Exp. Anal. Behav. 15:1971;237-241.
30. Daw N.D., Kakade S., Dayan P. Opponent interactions between serotonin and dopamine. Neural Networks. 15:2002;603-616.
31. Dayan P. Improving generalization for temporal difference learning. the successor representation Neural Comput. 5:1993;613-624.
32. Dayan P. Motivated reinforcement learning. Dietterich T.G., Becker S., Ghahramani Z. Advances in Neural Information Processing Systems, 2001. 2002;MIT Press, Cambridge, MA. in press.
33. Deakin J.F.W. Roles of brain serotonergic neurons in escape, avoidance and other behaviors. J. Psychopharmacol. 43:1983;563-577.
34. Deakin J.F.W., Graeff F.G. 5-HT and mechanisms of defence. J. Psychopharmacol. 5:1991;305-316.
35. DeBold R.C., Miller N.E., Jensen D.D. Effect of strength of drive determined by a new technique for appetitive classical conditioning of rats. J. Comp. Physiol. Psychol. 59:1965;102-108.
36. de Borchgrave R., Rawlins J.N.P., Dickinson A., Balleine B.W. The role of the nucleus accumbens in instrumental conditioning. Exp. Brain Res. 144:2002;50-68.
37. Di Chiara G., North R.A. Neurobiology of opiate abuse. Trends Pharmacol. Sci. 13:1992;185-193.
38. Dickinson A. Instrumental conditioning. Mackintosh N.J. Animal Learning and Cognition. 1994;45-79 Academic Press, San Diego.
39. Dickinson A. Bolles's psychological syllogism. Bouton M.E., Fanselow M.S. Learning, Motivation, and Cognition. The Functional Behaviorism of Robert C. Bolles:1997;345-367 American Psychological Association, Washington, DC.
40. Dickinson A., Balleine B.W. Actions and responses. the dual psychology of behaviour Eilan N., McCarthy R., Brewer M.W. Spatial Representation. 1993;277-293 Basil Blackwell Ltd, Oxford.
41. Dickinson A., Balleine B.W. Motivational control of goal-directed action. Anim. Learn. Behav. 22:1994;1-18.
42. Dickinson A., Balleine B.W. Motivational control of instrumental action. Current Directions in Psychological Science. 4:1995;162-167.
43. Dickinson A., Balleine B.W. The role of learning in motivation. Gallistel C.R. Learning, Motivation and Emotion, Volume 3 of Steven's Handbook of Experimental Psychology. Third Edition:2002;John Wiley & Sons, New York. in press.
44. Dickinson A., Dawson G.R. Pavlovian processes in the motivational control of instrumental performance. Q. J. Exp. Psychol. B. 39:1987;201-213.
45. Dickinson A., Dawson G.R. Motivational control of instrumental performance. the role of prior experience of the reinforcer Q. J. Exp. Psychol. B. 40:1988;113-134.
46. Dickinson A., Dearing M.F. Appetitive-aversive interactions and inhibitory processes. Dickinson A., Boakes R.A. Mechanism of Learning and Motivation. 1979;Lawrence Erlbaum Associates, Hillsdale, NJ. 203-231.pp.
47. Dickinson A., Mulatero C.W. Reinforcer specificity of the suppression of instrumental performance on a non-contingent schedule. Behavioral Processes. 19:1989;167-180.
48. Dickinson A., Balleine B.W., Watt A., Gonzalez F., Boakes R.A. Motivational control after extended instrumental training. Anim. Learn. Behav. 23:1995;197-206.
49. Dickinson A., Squire S., Varga Z., Smith J.W. Omission learning after instrumental pretraining. Q. J. Exp. Psychol. B. 51:1998;271-286.
50. Dickinson A., Smith J., Mirenowicz J. Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav. Neurosci. 114:2000;468-483.
51. Doyle T.G., Berridge K.C., Gosnell B.A. Morphine enhances hedonic taste palatability in rats. Pharmacol. Biochem. Behav. 46:1993;745-749.
52. Drewnowski A., Krahn D.D., Demitrack M.A., Nairn K., Gosnell B.A. Taste responses and preferences for sweet high-fat foods. evidence for opioid involvement Physiol. Behav. 51:1992;371-379.
53. Dworkin S.I., Guerin G.F., Goerders N.E., Smith J.E. Lack of an effect of 6-hydroxydopamine lesions of the nucleus accumbens on intravenous morphine self-administration. Pharmacol. Biochem. Behav. 30:1988;1051-1057.
54. Everitt B.J., Stacey P. Studies of instrumental behavior with sexual reinforcement in male rats (Rattus novegicus). II. effects of preoptic area lesions, castration, and testosterone J. Comp. Psychol. 101:1987;407-419.
55. Fudim O.K. Sensory preconditioning of flavors with formalin-produced sodium need. J. Exp. Psychol. Anim. Behav. Process. 4:1978;276-285.
56. Gerfen C.R. The neostriatal mosaic. I. compartmental organisation of projections from the striatum to the substantia nigra of the rat. J. Comp. Neurol. 236:1985;454-476.
57. Gerfen C.R. The neostriatal mosaic. multiple levels of compartmental organization Trends Neurosci. 15:1992;133-139.
58. Gerfen C.R., Engber T.M., Mahan L.C., Susel Z., Chase T.N., Monsma F.J., Sibley D.R. D1 and D2 dopamine receptor-regulated gene expression of striatonigral and striatopallidal neurons. Science. 250:1990;1429-1432.
59. Gluck M.A., Allen M.T., Myers C.E., Thompson R.F. Cerebellar substrates for error correction in motor conditioning. Neurobiol. Learn. Mem. 76:2001;314-341.
60. Gong W., Neil D., Justice J.B. Conditioned place preference and locomotor activation produced by injection of psychostimulants into ventral pallidum. Brain Res. 707:1996;64-74.
61. Gosnell B.A., Majchrzak M.J. Centrally administered opioid peptides stimulate saccharin intake in nondeprived rats. Pharmacol. Biochem. Behav. 33:1993;805-810.
62. Graybiel A.M., Aosaki T., Flaherty A.W., Kimura M. The basal ganglia and adaptive motor control. Science. 265:1994;1826-1831.
63. Grossberg S.E. Neural Networks and Natural Intelligence. 1988;MIT Press, Cambridge, MA.
64. Haber S.N., Fudge J.L., McFarland N.R. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 20:2000;2369-2382.
65. Hammond L.J. The effects of contingencies upon appetitive conditioning of free-operant behavior. J. Exp. Anal. Behav. 34:1980;297-304.
66. Hendersen R.W., Graham J. Avoidance of heat by rats. effects of thermal context on the rapidity of extinction Learn. Motiv. 10:1979;351-363.
67. Hershberger W.A. An approach through the looking glass. Anim. Learn. Behav. 14:1986;443-451.
68. Holland P.C. Conditioned stimulus as a determinant of the form of the Pavlovian conditioned response. J. Exp. Psychol. Anim. Behav. Process. 3:1977;77-104.
69. Holland P.C. Differential effects of omission contingencies on various components of Pavlovian appetitive responding in rats. J. Exp. Psychol. Anim. Behav. Process. 5:1979;178-193.
70. Holland P.C. Cognitive aspects of classical conditioning. Curr. Opin. Neurobiol. 3:1993;230-236.
71. Hollis K.L., Pharr V.L., Dumas M.J., Britton G.B., Field J. Classical conditioning provides paternity advantage for territorial male blue gouramis (Trichogaster trichopterus). J. Comp. Psychol. 111:1997;219-225.
72. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Houk J.C., Davis J.L. Models of Information Processing in the Basal Ganglia. 1995;MIT Press, Cambridge, MA. 249-270.pp.
73. Howland J.G., Taepavarapruk P., Phillips A.G. Glutamate receptor-dependent modulation of dopamine efflux in the nucleus accumbens by basolateral, but not central, nucleus of the amygdala in rats. J. Neurosci. 22:2002;1137-1145.
74. Johnson P.I., Stellar J.R., Paul A.D. Regional reward differences within the ventral pallidum are revealed by microinjections of a mu opiate receptor agonist. Neuropharmacology. 32:1993;1305-1314.
75. Kakade S., Dayan P. Dopamine bonuses. Leen T.K., Dietterich T.G., Tresp V. Advances in Neural Information Processing Systems 2000. 2001;MIT Press, Cambridge, MA. 131-137.pp.
76. Kakade S., Dayan P. Dopamine. generalization and bonuses Neural Networks. 15:2002;549-559.
77. Kearns M.S., Singh S. Finite-sample rates of convergence for Q-learning and indirect methods. Kearns M., Solla S.A., Cohn D. Advances in Neural Information Processing Systems, Volume 11. 1999;MIT Press, Cambridge, MA. 996-1002.pp.
78. Kelley A.E., Bless E.P., Swanson C.J. Investigation of the effects of opiate antagonists infused into the nucleus accumbens on feeding and sucrose drinking in rats. J. Pharmacol. Exp. Ther. 278:1996;1499-1507.
79. Konorski J. Integrative Activity of the Brain. An Interdisciplinary Approach:1967;University of Chicago Press, Chicago.
80. Krebs J.R., Kacelnik A., Taylor P. Test of optimal sampling by foraging great tits. Nature. 275:1978;27-31.
81. Krieckhaus E.E. "Innate recognition" aids rats in sodium regulation. J. Comp. Physiol. Psychol. 73:1970;117-122.
82. Lopez M., Paredes-Olay C. Sensitivity of instrumental responses to an upshift in water deprivation. Anim. Learn. Behav. 27:1999;280-287.
83. Lopez M., Balleine B., Dickinson A. Incentive learning and the motivational control of instrumental performance by thirst. Anim. Learn. Behav. 20:1992;322-328.
84. Mackintosh N.J. The Psychology of Animal Learning. 1974;Academic Press, New York.
85. Mangel M., Clark C.W. Dynamic Modeling in Behavioral Ecology. 1988;Princeton University Press, Princeton.
86. Martin del Campo A.F., Dowson J.H., Herbert J., Paykel E.S. Effects of naloxone on diurnal rhythms in mood and endocrine function. a dose response study Psychopharmacology (Berl.). 114:1994;583-590.
87. Mitchell D.S., Gormezano I. Effects of water deprivation on classical appetitive conditioning of the rabbit's jaw movement response. Learn. Motiv. 1:1970;199-206.
88. Montague P.R., Berns G.S. Neural economics and biological substrates of valuation. Neuron. 36:2002;265-284. this issue,
89. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16:1996;1936-1947.
90. O'Brien C.P., Childress A.R., Ehrman R., Robbins S.J. Conditioning factors in drug abuse. can they explain compulsion? J. Psychopharmacol. 12:1998;15-22.
91. Parker L.A., Maier S., Rennie S.M., Crebolder J. Morphine- and naltrexone-induced modification of palatability. analysis by the taste reactivity test Behav. Neurosci. 106:1992;999-1010.
92. Pecina S., Berridge K.C. Opioid site in nucleus accumbens shell mediates eating and hedonic 'liking' for food. map based on microinjection Fos plumes Brain Res. 863:2000;71-86.
93. Pecina S., Berridge K.C., Parker L.A. Pimozide does not shift palatability. separation of anhedonia from sensorimotor effects Pharmacol. Biochem. Behav. 58:1997;801-811.
94. Pettit H.O., Ettenberg A., Bloom F.E., Koob G.F. Destruction of dopamine in the nucleus accumbens selectively attenuates cocaine but not heroin self-administration in rats. Psychopharmacology (Berl.). 84:1984;167-173.
95. Phillips A.G., LePiane F.G., Fibiger H.C. Dopaminergic mediation of reward produced by direct injection of enkephalin into the ventral tegmental area in the rat. Life Sci. 33:1983;2505-2511.
96. Rescorla R.A. Probability of shock in the presence and absence of CS in fear conditioning. J. Comp. Physiol. Psychol. 66:1968;1-5.
97. Rescorla R.A. Simultaneous second-order conditioning produces S-S learning in conditioned suppression. J. Exp. Psychol. Anim. Behav. Process. 8:1982;23-32.
98. Rescorla R.A., Solomon R.L. Two-process learning theory. relationships between Pavlovian conditioning and instrumental learning Psychol. Rev. 74:1967;151-182.
99. Reynolds N.J., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature. 413:2001;67-70.
100. Rolls E.T. Information processing in the taste system of primates. J. Exp. Biol. 146:1989;141-164.
101. Rolls E.T. The orbitofrontal cortex and reward. Cereb. Cortex. 10:2000;284-294. a.
102. Rolls E.T. Memory systems in the brain. Annu. Rev. Psychol. 51:2000;599-630. b.
103. Rolls E.T., Yaxley S., Sienkiewicz Z.J. Gustatory responses of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey. J. Neurophysiol. 64:1990;1055-1066.
104. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80:1998;1-27.
105. Schultz W. Getting formal with dopamine and reward. Neuron. 36:2002;241-263. this issue,
106. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science. 275:1997;1593-1599.
107. Solomon R.L., Corbit J.D. An opponent-process theory of motivation. I. Temporal dynamics of affect. Psychol. Rev. 81:1974;119-145.
108. Suri R.E. TD models of reward predictive responses in dopamine neurons. Neural Networks. 15:2002;523-533.
109. Suri R.E., Schultz W. Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121:1998;350-354.
110. Suri R.E., Schultz W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience. 91:1999;871-890.
111. Suri R.E., Schultz W. Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13:2001;841-862.
112. Sutton R.S., Barto A.G. Reinforcement Learning. An Introduction:1998;MIT Press, Cambridge, MA.
113. Sutton, R.S., and Pinette, B. (1985). The learning of world models by connectionist networks. In Proceedings of the Seventh Annual Conference of the Cognitive Science Society (Irvine, CA: Lawrence Erlbaum), pp. 54-64.
114. Taylor J.R., Robbins T.W. Enhanced behavioral control by conditioned reinforcers following micro-injections of d-amphetamine into the nucleus accumbens. Psychopharmacology (Berl.). 84:1984;405-412.
115. Toates F.M. Motivational Systems. 1986;Cambridge University Press, Cambridge, UK.
116. Toates F.M. Comparing motivational systems. an incentive motivation perspective Legg C.R., Booth D.A. Appetite. Neural and Behavioural Bases:1994;305-327 Oxford University Press, New York.
117. Wagner A.R., Brandon S.E. Evolution of a structured connectionist model of Pavlovian conditioning (AESOP). Klein S.B., Mowrer R.R. Contemporary learning theories. 1989;Lawrence Erlbaum Associates, Hillsdale, NJ. 149-189.pp.
118. Watkins, C.J.C.H. (1989). Learning from delayed rewards. PhD thesis, University of Cambridge, Cambridge, United Kingdom.
119. Wyvell C.L., Berridge K.C. Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward. enhancement of reward "wanting" without enhanced liking or response reinforcement J. Neurosci. 20:2000;8122-8130.
120. Wyvell C.L., Berridge K.C. Incentive sensitization by previous amphetamine exposure. increased cue-triggered "wanting" for sucrose reward J. Neurosci. 21:2001;7831-7840.
121. Wolpert D.M., Ghahramani Z. Computational principles of movement neuroscience. Nat. Neurosci. 3:2000;1212-1217.
122. Yaxley S., Rolls E.T., Sienkiewicz Z.J. The responsiveness of neurons in the insular gustatory cortex of the macaque monkey is independent of hunger. Physiol. Behav. 42:1988;223-229.