Advanced Reinforcement Learning

Neuroeconomics: Decision Making and the Brain: Second Edition

Volumn , Issue , 2013, Pages 299-320

  Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

Dopamine; Hierarchical reinforcement learning; Reinforcement learning; Uncertainty

Indexed keywords

EID: 84897397355     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-416008-8.00016-4     Document Type: Chapter

References (131)

1. Alexander G.E., Crutcher M.D. Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 1990, 13:266-271.
2. Arbuthnott G.W., Wickens J. Space, time and dopamine. Trends Neurosci. 2007, 30:62-69.
3. Badre D., Frank M.J. Mechanisms of hierarchical reinforcement learning in cortico-striatal circuits 2: evidence from fMRI. Cereb. Cortex 2012, 22:527-536.
4. Badre D., Kayser A.S., D'Esposito M. Frontal cortex and the discovery of abstract action rules. Neuron 2010, 66:315-326.
5. Barto A.G. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
6. Barto A.G., Mahadevan S. Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst. 2003, 13:341-379.
7. Barto A.G., Sutton R.S., Brouwer P.S. Associative search network - a reinforcement learning associative memory. Biol. Cybern. 1981, 40(3):201-211.
8. Bayer H.M., Lau B., Glimcher P.W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 2007, 98:1428-1439.
9. Behrens T., Woolrich M., Walton M., Rushworth M. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
10. Bellman R. Dynamic Programming 1957, Princeton University Press, Princeton.
11. Berridge K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 2007, 191:391-431.
12. Bertsekas D.P., Tsitsiklis J.N. Neuro-Dynamic Programming 1996, Athena Scientific, Belmont, Mass.
13. Bogacz R., McClure S.M., Li J., Cohen J.D., Montague P.R. Short-term memory traces for action bias in human reinforcement learning. Brain Res. 2007, 1153:111-121.
14. Bornstein A.M., Daw N.D. Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr. Opin. Neurobiol. 2011, 21:374-380.
15. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
16. Brinkman C. Supplementary motor area of the monkey's cerebral cortex: short-and long-term deficits after unilateral ablation and the effects of subsequent callosal section. J. Neurosci. 1984, 4:918-929.
17. Brischoux F., Chakraborty S., Brierley D.I., Ungless M.A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc. Natl. Acad. Sci. U.S.A. 2009, 106:4894-4899.
18. Caplin A., Dean M. Dopamine, reward prediction error, and economics. Q. J. Econ. 2008, 123:663-701.
19. Chang Y.H., Ho T., Kaelbling L.P. All learning is local: Multi-agent learning in global reward games. Adv. Neural Inf. Process. Syst. 2003, 16:807-814.
20. Charnov E.L. Optimal foraging, the marginal value theorem. Theor. Popul. Biol. 1976, 9:129-136.
21. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
22. Courville A.C., Daw N.D., Gordon G.J., Touretzky D.S. Model uncertainty in classical conditioning. Adv. Neural Inf. Process. Syst. 2003, 16:977-984.
23. Courville A.C., Daw N.D., Touretzky D.S. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 2006, 10:294-300.
24. D'amato M., Fazzaro J., Etkin M. Anticipatory responding and avoidance discrimination as factors in avoidance conditioning. J. Exp. Psychol. 1968, 77:41.
25. Daw N.D., Courville A.C., Touretzky D.S. Representation and timing in theories of the dopamine system. Neural Comput. 2006, 18:1637-1677.
26. Daw N.D., Kakade S., Dayan P. Opponent interactions between serotonin and dopamine. Neural Netw. 2002, 15:603-616.
27. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
28. Dayan P., Daw N.D. Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 2008, 8:429-453.
29. Dayan P., Kakade S., Montague P.R. Learning and selective attention. Nat. Neurosci. 2000, 3:1218-1223.
30. Dayan P., Long T. Statistical models of conditioning. Adv. Neural Inf. Process. Syst. 1998, 117-123.
31. Dearden R., Friedman N., Russell S., 1998. Bayesian Q-learning. In: John Wiley & Sons Ltd, pp. 761-768.
32. Dezfouli A., Balleine B.W. Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 2012, 35:1036-1051.
33. Dommett E., Coizet V., Blaha C.D., Martindale J., Lefebvre V., Walton N., et al. How visual stimuli activate dopaminergic neurons at short latency. Sci. Signal. 2005, 307:1476.
34. Floresco S.B., West A.R., Ash B., Moore H., Grace A.A. Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nat. Neurosci. 2003, 6:968-973.
35. Frank M.J., Badre D. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis. Cereb. Cortex 2012, 22:509-526.
36. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 2004, 306:1940-1943.
37. Gallistel C., Stellar J.R., Bubis E. Parametric analysis of brain stimulation reward in the rat: I. The transient process and the memory-containing process. J. Comp. Physiol. Psychol. 1974, 87:848.
38. Garris P.A., Christensen J.R.C., Rebec G.V., Wightman R.M. Real-time measurement of electrically evoked extracellular dopamine in the striatum of freely moving rats. J. Neurochem. 2002, 68:152-161.
39. Gershman S., Pesaran B., Daw N. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J. Neurosci. 2009, 29:13524-13531.
40. Gershman S.J., Blei D.M., Niv Y. Context, learning, and extinction. Psychol. Rev. 2010, 117:197.
41. Gershman S.J., Cohen J.D., Niv, Y., 2010b. Learning to selectively attend. Proceedings of the 32nd Annual Conference of the Cognitive Science Society, pp. 1270-1275.
42. Gershman S.J., Niv Y. Learning latent structure: carving nature at its joints. Curr. Opin. Neurobiol. 2010, 20:251.
43. Gold J., Shadlen M. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 2002, 36:299-308.
44. Guitart-Masip M., Beierholm U.R., Dolan R., Duzel E., Dayan P. Vigor in the face of fluctuating rates of reward: an experimental examination. J. Cogn. Neurosci. 2011, 23:3933-3938.
45. Haber S.N., Fudge J.L., McFarland N.R. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 2000, 20:2369-2382.
46. Haith A.M., Reppert T.R., Shadmehr R. Evidence for hyperbolic temporal discounting of reward in control of movements. J. Neurosci. 2012, 32:11727-11736.
47. Helmholtz H. Handbuch der Physiologischen Optik 1860, Leopold Voss, Leipzig.
48. Holland P.C. Brain mechanisms for changes in processing of conditioned stimuli in Pavlovian conditioning: Implications for behavior theory. Learn. Behav. 1997, 25:373-399.
49. Horvitz J. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 2000, 96:651-656.
50. Houk, J.C., Adams, J.L., Barto, A.G., 1995. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk, J.C., Davis, J.L., Beiser, D.G. (Eds.) Models of Information Processing in the Basal Ganglia. MIT Press, Boston, pp. 249-270.
51. Izhikevich E.M. Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 2007, 17(10):2443-2452.
52. Jog M.S., Kubota Y., Connolly C.I., Hillegaart V., Graybiel A.M. Building neural representations of habits. Science 1999, 286:1745-1749.
53. Joshua M., Adler A., Mitelman R., Vaadia E., Bergman H. Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J. Neurosci. 2008, 28:11673-11684.
54. Kaelbling L.P., Littman M.L., Cassandra A.R. Planning and acting in partially observable stochastic domains. Artif. Intell. 1998, 101:99-134.
55. Kakade S., Dayan P. Acquisition and extinction in autoshaping. Psychol. Rev; Psychol. Rev. 2002, 109:533.
56. Kalman R.E. A new approach to linear filtering and prediction problems. J. Basic Eng. 1960, 82:35-45.
57. Killeen P.R. Models of trace decay, eligibility for reinforcement, and delay of reinforcement gradients, from exponential to hyperboloid. Behav. Process. 2011, 87(1):57-63.
58. Knill D.C., Pouget A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 2004, 27:712-719.
59. Konorski, J., 1967. Integrative activity of the brain. Leopold Voss, Leipzig.
60. Laplane D., Talairach J., Meininger V., Bancaud J., Orgogozo J. Clinical consequences of corticectomies involving the supplementary motor area in man. J. Neurol. Sci. 1977, 34:301-314.
61. Lewicki M.S., Olshausen B.A. Probabilistic framework for the adaptation and comparison of image codes. JOSA A 1999, 16:1587-1601.
62. Li J., Schiller D., Schoenbaum G., Phelps E.A., Daw N.D. Differential roles of human striatum and amygdala in associative learning. Nat. Neurosci. 2011, 14:1250-1252.
63. Loch J., Singh S., 1998. Using eligibility traces to find the best memoryless policy in partially observable markov decision processes. In: Proceedings of the 15th International Conference on Machine Learning. Morgan Kaufmann, San Francisco, CA.
64. Lyon M., Robbins T. The action of central nervous system stimulant drugs: a general theory concerning amphetamine effects. Curr. Dev. Psychopharmacol. 1975, 2:79-163.
65. Ma W.J., Beck J.M., Latham P.E., Pouget A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 2006, 9:1432-1438.
66. Maia T. Two-factor theory, the actor-critic model, and conditioned avoidance. Learn Behav. 2010, 38:50-67.
67. Margolis E.B., Lock H., Hjelmstad G.O., Fields H.L. The ventral tegmental area revisited: is there an electrophysiological marker for dopaminergic neurons?. J. Physiol. 2006, 577:907-924.
68. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
69. Mazzoni P., Hristova A., Krakauer J.W. Why don't we move faster? Parkinson's disease, movement vigor, and implicit motivation. J. Neurosci. 2007, 27:7105-7116.
70. McKinstry J.L., Edelman G.M., Krichmar J.L. A cerebellar model for predictive motor control tested in a brain-based device. Proc. Natl. Acad. Sci. U.S.A 2006, 103(9):3387-3392.
71. Mirenowicz J., Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 1996, 379:449-451.
72. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16:1936-1947.
73. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
74. Moutoussis M., Bentall R.P., Williams J., Dayan P. A temporal difference account of avoidance learning. Network: Comput. Neural Syst. 2008, 19:137-160.
75. Mowrer O.H. Two-factor learning theory: summary and comment. Psychol. Rev. 1951, 58:350.
76. Newsome W.T., Pare E.B. A selective impairment of motion perception following lesions of the middle temporal visual area (MT). J. Neurosci. 1988, 8:2201-2211.
77. Niv Y., Daw N., Dayan P. How fast to work: response vigor, motivation and tonic dopamine. Adv. Neural Inf. Process. Syst. 2006, 18:1019.
78. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl) 2007, 191:507-520.
79. Nomoto K., Schultz W., Watanabe T., Sakagami M. Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J. Neurosci. 2010, 30:10692-10702.
80. Olshausen B.A. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 1996, 381:607-609.
81. Palminteri S., Boraud T., Lafargue G., Dubois B., Pessiglione M. Brain hemispheres selectively track the expected value of contralateral options. J. Neurosci. 2009, 29:13465-13472.
82. Pan W.-X., Schmidt R., Wickens J.R., Hyland B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 2005, 25(26):6235-6242.
83. Parr R., Russell S. Reinforcement learning with hierarchies of machines. Adv. Neural Inf. Process. Syst. 1998, 1043-1049.
84. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532.
85. Phillips P.E., Stuber G.D., Heien M.L., Wightman R.M., Carelli R.M. Subsecond dopamine release promotes cocaine seeking. Nature 2003, 422:614-618.
86. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 1999, 400:233-238.
87. Preuschoff K., Bossaerts P. Adding prediction risk to the theory of reward learning. Ann. N. Y. Acad. Sci. 2007, 1104:135-146.
88. Puterman M.L. Markov Decision Processes: Discrete Stochastic Dynamic Programming 1994, John Wiley & Sons, New York.
89. Rao R.P.N. Decision making under uncertainty: a neural model based on partially observable markov decision processes. Front. Comput. Neurosci. 2010, 4:146.
90. Rescorla R., Wagner A. Variations in the Effectiveness of Reinforcement and Nonreinforcement 1972, Classical Conditioning II: Current Research and Theory, Appleton-Century-Crofts, New York.
91. Reynolds J.N., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 2002, 15:507-521.
92. Ribas-Fernandes J.J.F., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., et al. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71:370-379.
93. Robbins T.W., Everitt B.J. A role for mesencephalic dopamine in activation: commentary on Berridge (2006). Psychopharmacology (Berl) 2007, 191:433-437.
94. Roesch M.R., Calu D.J., Esber G.R., Schoenbaum G. Neural correlates of variations in event processing during learning in basolateral amygdala. J. Neurosci. 2010, 30:2464-2471.
95. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 2007, 10:1615-1624.
96. Roitman J.D., Shadlen M.N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 2002, 22:9475-9489.
97. Rothkopf C.A., Ballard D.H. Credit assignment in multiple goal embodied visuomotor behavior. Front. Psychol. 2010, 1:173.
98. Rummery, G., Niranjan, M., 1994. On-line Q-learning using connectionist systems, Cambridge University.
99. Russell, S., Zimdars, A.L., 2003. Q-decomposition for reinforcement learning agents. Proceedings of ICML-03.
100. Rutledge R.B., Dean M., Caplin A., Glimcher P.W. Testing the reward prediction error hypothesis with an axiomatic model. J. Neurosci. 2010, 30:13525-13536.
101. Rutledge R.B., Lazzaro S.C., Lau B., Myers C.E., Gluck M.A., Glimcher P.W. Dopaminergic drugs modulate learning rates and perseveration in Parkinson's patients in a dynamic foraging task. J. Neurosci. 2009, 29:15104-15114.
102. Salamone J.D., Correa M., Farrar A., Mingote S.M. Effort-related functions of nucleus accumbens dopamine and associated forebrain circuits. Psychopharmacology 2007, 191:461-482.
103. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
104. Seo H., Barraclough D.J., Lee D. Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. Cereb. Cortex 2007, 17(Suppl. 1):i110-117.
105. Singh S.P., Jaakkola T., Jordan M.I., 1994. Learning without state-estimation in partially observable markovian decision processes. In: International Conference on Machine Learning.
106. Solomon R.L., Corbit J.D. An opponent-process theory of motivation: I. Temporal dynamics of affect. Psychol. Rev. 1974, 81:119.
107. Stephens D.W., Krebs J.R. Foraging Theory 1987, Princeton University Press, Princeton, NJ.
108. Sutton R.S. Learning to predict by the methods of temporal differences. Mach. Learn. 1988, 3:9-44.
109. Sutton, R.S., 1992. Gain adaptation beats least squares? In: Proceedings of the Seventh Yale Workshop on Adaptive and Learning Systems. Yale University, New Haven, CT, pp. 161-166.
110. Sutton, R.S., 1995. TD models: modeling the world at a mixture of time scales. In: International Conference on Machine Learning, pp. 531-539.
111. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
112. Sutton R.S., Precup D., Singh S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artif. Intell. 1999, 112:181-211.
113. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., et al. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci 2011, 14(12):1590-1597.
114. Tanji J., Okano K., Sato K.C. Neuronal activity in cortical motor areas related to ipsilateral, contralateral, and bilateral digit movements of the monkey. J. Neurophysiol. 1988, 60:325-343.
115. Tenenbaum J.B., Griffiths T.L. Structure learning in human causal induction. Adv. Neural Inf. Process. Syst. 2001, 13:59-65.
116. Thorn C.A., Atallah H., Howe M., Graybiel A.M. Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 2010, 66:781-795.
117. Tobler P.N., Fiorillo C.D., Schultz W. Adaptive coding of reward value by dopamine neurons. Science 2005, 307:1642-1645.
118. Todd M.T., Niv Y., Cohen J.D., 2009. Learning to use working memory in partially observable environments through dopaminergic reinforcement. In: Advances in Neural Information Processing Systems 21.
119. Tsai H.C., Zhang F., Adamantidis A., Stuber G.D., Bonci A., de Lecea L., et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 2009, 324:1080-1084.
120. Ungless M.A., Grace A.A. Are you or aren't you? Challenges associated with physiologically identifying dopamine neurons. Trends Neurosci 2012, 35(7):422-430.
121. Ungless M.A., Magill P.J., Bolam J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 2004, 303:2040-2042.
122. von Neumann J.V., Morgenstern O. Theory of Games and Economic Behavior 1947, Princeton University Press, Princeton, NJ. second ed.
123. Wang S.S., Denk W., Hausser M. Coincidence detection in single dendritic spines mediated by calcium release. Nat. Neurosci. 2000, 3(12):1266-1273.
124. Wang L.P., Li F., Wang D., Xie K., Shen X., Tsien J.Z. NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron 2011, 72:1055-1066.
125. Watkins C.J.C.H., Dayan P. Q-learning. Mach. Lear. 1992, 8:279-292.
126. Wickens J.R., Kötter R. Cellular models of reinforcement. Models of Information Processing in the Basal Ganglia 1995, 187-214. MIT Press. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
127. Wilson R.C., Niv Y. Inferring relevance in a changing world. Front. Human Neurosci. 2011, 5:189.
128. Wunderlich K., Rangel A., O'Doherty J.P. Neural computations underlying action-based decision making in the human brain. Proc. Natl. Acad. Sci. U.S.A. 2009, 106:17199-17204.
129. Yang T., Shadlen M.N. Probabilistic reasoning by neurons. Nature 2007, 447:1075-1080.
130. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.
131. Yuille A., Kersten D. Vision as Bayesian inference: analysis by synthesis?. Trends Cogn. Sci. 2006, 10:301-308.