Value Learning through Reinforcement: The Basics of Dopamine and Reinforcement Learning

Neuroeconomics: Decision Making and the Brain: Second Edition

Volumn , Issue , 2013, Pages 283-298

  Daw, Nathaniel D ^a   Tobler, Philipe N ^b  

^a NEW YORK UNIVERSITY   (United States)

^b UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Dopamine; Reinforcement learning

Indexed keywords

EID: 84897401223     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-416008-8.00015-2     Document Type: Chapter

References (81)

1. Abler B., Walter H., Erk S., Kammerer H., Spitzer M. Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 2006, 31:790-795.
2. Adamantidis A.R., Tsai H.C., Boutrel B., Zhang F., Stuber G.D., Budygin E.A., et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 2011, 31:10829-10835.
3. Alexander G.E., Crutcher M.D. Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 1990, 13:266-271.
4. Andén N.E., Fuxe K., Hamberger B., Hökfelt T. A quantitative study on the nigro-neostriatal dopamine neurons. Acta Physiol. Scand. 1966, 67:306-312.
5. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
6. Bayer H.M., Lau B., Glimcher P.W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 2007, 98:1428-1439.
7. Bellman R. Dynamic Programming 1957, Princeton University Press, Princeton, NJ.
8. Bodi N., Keri S., Nagi H., Moustafa A., Myers C., Daw N.D., et al. Reward learning and the novelty seeking personality: a between and within-subjects study of the effects of dopamine agonists on young Parkinson's patients. Brain 2009, 2132:2385-2395.
9. Bush R.R., Mosteller F. A mathematical model for simple learning. Psychol. Rev. 1951, 58:313-323.
10. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
11. Cools R., Altamirano L., D'Esposito M. Reversal learning in Parkinson's disease depends on medication status and outcome valence. Neuropsychologia 2006, 44:1663-1673.
12. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
13. Day J.J., Roitman M.F., Wightman R.M., Carelli R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 2007, 10:1020-1028.
14. DeLong M.R. Primate models of movement disorders of basal ganglia origin. Trends Neurosci. 1990, 13:281-285.
15. Düzel E., Bunzeck N., Guitart-Masip M., Wittmann B., Schott B.H., Tobler P.N. Functional imaging of the human dopaminergic midbrain. Trends Neurosci. 2009, 32:321-328.
16. Fiorillo C.D., Tobler P.N., Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 2003, 299:1898-1902.
17. Felten D., Shetty A. Netter's Atlas of Neuroscience 2009, Philadelphia Saunders-Elsevier.
18. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
19. Freund T.F., Powell J.F., Smith A.D. Tyrosine hydroxylase-immunoreactive boutons in synaptic contact with identified striatonigral neurons, with particular reference to dendritic spines. Neuroscience 1984, 13:1189-1215.
20. Fu W.T., Anderson J.R. Solving the credit assignment problem: explicit and implicit learning of action sequences with probabilistic outcomes. Psychol. Res. 2008, 72:321-330.
21. Gallistel C.R., Stellar J., Bubis E. Parametric analysis of brain stimulation reward in the rat: I. The transient process and the memory-containing process. J. Comp. Physiol. Psychol. 1974, 87:848-860.
22. German D.C., Dubach M., Askari S., Speciale S.G., Bowden D.M. 1-Methyl-4-phenyl-1,2,3,6-tetrahydropyridine-induced parkinsonian syndrome in Macaca fascicularis: which midbrain dopaminergic neurons are lost?. Neuroscience 1988, 24:161-174.
23. Glimcher P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. 2011, 108(Suppl. 3):15647-15654.
24. Grace A.A., Bunney B.S. Intracellular and extracellular electrophysiology of nigral dopaminergic neurons-3. Evidence for electrotonic coupling. Neuroscience 1983, 10:333-348.
25. Groves P.M., Linder J.C., Young S.J. 5-hydroxydopamine-labeled dopaminergic axons: three-dimensional reconstructions of axons, synapses and postsynaptic targets in rat neostriatum. Neuroscience 1994, 58:593-604.
26. Haber S.N., Knutson B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 2010, 35:4-26.
27. Hollerman J.R., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1998, 1:304-309.
28. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of information processing in the basal ganglia 1995, 249-270. MIT Press, Boston. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
29. Hyman S.E., Malenka R.C. Addiction and the brain: the neurobiology of compulsion and its persistence. Nat. Rev. Neurosci. 2001, 2:695-703.
30. Ito M., Doya K. Validation of decision-making models and analysis of decision variables in the rat basal ganglia. J. Neurosci. 2009, 29:9861-9874.
31. Kamin L.J. Predictability, surprise, attention and conditioning. Punishment and Aversive Behavior 1969, 279-296. Appleton-Century-Crofts, New York. B.A. Campbell, R.M. Church (Eds.).
32. Kawagoe R., Takikawa Y., Hikosaka O. Reward-predicting activity of dopamine and caudate neurons-a possible mechanism of motivational control of saccadic eye movement. J. Neurophysiol. 2004, 91:1013-1024.
33. Kim K.M., Baratta M.V., Yang A., Lee D., Boyden E.S., Fiorillo C.D. Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement. PLoS One 2012, 7:e33612.
34. Knutson B., Gibbs S.E.B. Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology 2007, 191:813-822.
35. Lau B., Glimcher P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 2005, 84:555-579.
36. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
37. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
38. Mirenowicz J, Schultz W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 1994, 72:1024-1027.
39. Mogenson G.J., Jones D.L., Yim C.Y. From motivation to action: functional interface between the limbic system and the motor system. Prog. Neurobiol. 1980, 14:69-97.
40. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16:1936-1947.
41. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
42. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 2004, 41:269-280.
43. Niv Y. Reinforcement learning in the brain. J. Math. Psychol. 2009, 53:139-154.
44. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 2007, 191:507-520.
45. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
46. Oyama K., Hernádi I., Iijima T., Tsutsui K. Reward prediction error coding in dorsal striatal neurons. J. Neurosci. 2010, 30:11447-11457.
47. Pavlov I.P. Conditional Reflexes 1927/1960, Dover Publications, New York, (The 1960 edition is an unaltered republication of the 1927 translation by Oxford University Press).
48. Pessiglione M., Seymour B., Flandin G., Dolan R.J., Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
49. Redish A.D. Addiction as a computational process gone awry. Science 2004, 306:1944-1947.
50. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton Century Crofts, New York. A.H. Black, W.F. Prokasy (Eds.).
51. Reynolds J.N., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 2002, 15:507-521.
52. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 2007, 10:1615-1624.
53. Romer D.H. Do firms maximize? Evidence from professional football. J. Pol. Econ. 2006, 114:340-365.
54. Rutledge RB, Dean M, Caplin A, Glimcher PW. Testing the reward prediction error hypothesis with an axiomatic model. J. Neurosci. 2010, 30:13525-13536.
55. Rutledge R.B., Lazzaro S.C., Lau B., Myers C.E., Gluck M.A., Glimcher P.W. Dopaminergic drugs modulate learning rates and perseveration in Parkinson's patients in a dynamic foraging task. J. Neurosci. 2009, 29:15104-15114.
56. Satoh T., Nakai S., Sato T., Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 2003, 23:9913-9923.
57. Schonberg T., O'Doherty J.P., Joel D., Inzelberg R., Segev Y., Daw N.D. Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. Neuroimage 2010, 49:772-781.
58. Schultz W. Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J. Neurophysiol. 1986, 56:1439-1461.
59. Schultz W. Multiple dopamine functions at different time courses. Annu. Rev. Neurosci. 2007, 30:259-288.
60. Schultz W., Apicella P., Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 1993, 13:900-913.
61. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
62. Seymour B., O'Doherty J.P., Dayan P., Koltzenburg M., Jones A.K., Dolan R.J., et al. Temporal difference models describe higher-order learning in humans. Nature 2004, 429:664-667.
63. Seymour B., Daw N.D., Roiser J.P., Dayan P., Dolan R. Serotonin selectively modulates reward value in human decision-making. J. Neurosci. 2012, 32:5833-5842.
64. Spicer J., Galvan A., Hare T.A., Voss H., Glover G., Casey B. Sensitivity of the nucleus accumbens to violations in expectation of reward. Neuroimage 2007, 34:455-461.
65. Sugrue L.P., Corrado G.S., Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 2004, 304:1782-1787.
66. Sutton R.S. Learning to predict by the method of temporal difference. Mach. Learn. 1988, 3:9-44.
67. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
68. Takikawa Y., Kawagoe R., Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. J. Neurophysiol. 2004, 92:2520-2529.
69. Tan K.R., Yvon C., Turiault M., Mirzabekov J.J., Doehner J., Labouèbe G., et al. GABA neurons of the VTA drive conditioned place aversion. Neuron 2012, 73:1173-1183.
70. Thorndike E.L. Animal Intelligence: Experimental Studies 1911, Macmillan, New York.
71. Tobler P.N. Behavioral functions of dopamine neurons. Dopamine Handbook 2009, 316-330. Oxford University Press, Oxford. A. Björklund, S. Dunnett, L. Iversen, S. Iversen (Eds.).
72. Tobler P.N., Dickinson A., Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 2003, 23:10402-10410.
73. Tobler P.N., Fiorillo C.D., Schultz W. Adaptive coding of reward value by dopamine neurons. Science 2005, 307:1642-1645.
74. Tobler P.N., O'Doherty J.P., Dolan R.J., Schultz W. Human neural learning depends on reward prediction errors in the blocking paradigm. J. Neurophysiol. 2006, 95:301-310.
75. Tobler P.N., O'Doherty J.P., Dolan R.J., Schultz W. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol. 2007, 97:1621-1632.
76. Tsai H.C., Zhang F., Adamantidis A., Stuber G.D., Bonci A., de Lecea L., et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 2009, 324:1080-1084.
77. Vandecasteele M., Glowinski J., Venance L. Electrical synapses between dopaminergic neurons of the substantia nigra pars compacta. J. Neurosci. 2005, 25:291-298.
78. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 2001, 412:43-48.
79. Wise R.A. Neuroleptics and operant behavior: the anhedonia hypothesis. Behav. Brain Sci. 1982, 5:39-53.
80. Wise R.A. Neurobiology of addiction. Curr. Opin. Neurobiol. 1996, 6:243-251.
81. Zaghloul K.A., Blanco J.A., Weidemann C.T., McGill K., Jaggi J.L., Baltuch G.H., et al. Human substantia nigra neurons encode unexpected financial rewards. Science 2009, 323:1496-1499.