Theoretical and empirical studies of learning

Neuroeconomics

Volumn , Issue , 2009, Pages 331-351

  Niv, Yael ^a   Montague, P Read ^b  

^a PRINCETON UNIVERSITY   (United States)

^b BAYLOR COLLEGE OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84882559330     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/B978-0-12-374176-9.00022-1     Document Type: Chapter

References (133)

1. Baird, L.C. (1995). Residual algorithms: reinforcement learning with function approximation. In: A. Prieditis and S. Russell (eds), Proceedings of the 12th International Conference on Machine Learning (IMLL 95). San Mateo, CA: Morgan Kaufman, pp. 30-37.
2. Barto A.G. Adaptive critic and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
3. Barto A.G., Sutton R.S., Anderson C.W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Systems Man Cyber. 1983, 13:834-846.
4. Barto A.G., Sutton R.S., Watkins C.J.C.H. Sequential decision problems and neural networks. Advances in Neural Information Processing Systems 2 1989, 686-693. MIT Press, Cambride, MA. D.S. Touretzky (Ed.).
5. Barto A.G., Sutton R.S., Watkins C.J.C.H. Learning and sequential decision making. Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990, 539-602. MIT Press, Cambridge, MA. M. Gabriel, J. Moore (Eds.).
6. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
7. Bayer H.M., Lau B., Glimcher P.W. Statistics of midbrain dopamine neuron spike trains in the awake primate. J. Neurophysiol. 2007, 98:1428-1439.
8. Behrens T.E.J., Woolrich M.W., Walton M.E., Rushworth M.F.S. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
9. Bellman R.E. Dynamic Programming 1957, Princeton University Press, Princeton, NJ.
10. Bergstrom B.P., Garris P.A. "Passive stabilization" of striatal extracellular dopamine across the lesion spectrum encompassing the presymptomatic phase of Parkinson's disease: a voltammetric study in the 6-OHDA lesioned rat. J. Neurochem. 2003, 87:1224-1236.
11. Berns G.S., McClure S.M., Pagnoni G., Montague P.R. Predictability modulates human brain response to reward. J. Neurosci. 2001, 21:2793-2798.
12. Berridge K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacol. (Berl.) 2007, 191:391-431.
13. Berridge K.C., Robinson T.E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Res. Rev. 1998, 28:309-369.
14. Bertsekas D.P., Tsitsiklis J.N. Neuro-dynamic Programming 1996, Athena, London.
15. Bush R.R., Mosteller F. A mathematical model for simple learning. Psychol. Rev. 1951, 58:313-323.
16. Christoph G.R., Leonzio R.J., Wilcox K.S. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. J. Neurosci. 1986, 6:613-619.
17. Coizet V., Dommett E.J., Redgrave P., Overton P.G. Nociceptive responses of midbrain dopaminergic neurones are modulated by the superior colliculus in the rat. Neuroscience 2006, 139:1479-1493.
18. Daw N.D., Touretzky D.S. Behavioral results suggest an average reward TD model of dopamine function. Neurocomputing 2000, 32:679-684.
19. Daw N.D., Touretzky D.S. Long-term reward prediction in TD models of the dopamine system. Neural Computation 2002, 14:2567-2583.
20. Daw N.D., Kakade S., Dayan P. Opponent interactions between serotonin and dopamine. Neural Networks 2002, 15:603-616.
21. Daw N.D., O'Doherty J.P., Dayan P., et al. Cortical substrates for exploratory decisions in humans. Nature 2006, 441:876-879.
22. Day J.J., Roitman M.F., Wightman R.M., Carelli R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 2007, 10:1020-1028.
23. Dayan P., Abbott L.F. Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems 2001, MIT Press, Cambridge, MA.
24. Dayan P., Kakade S. Explaining away in weight space. Advances in Neural Information Processing Systems 2000, Vol. 12:24-30. MIT Press, Cambridge, MA. T. Leen, T. Dietterich, V. Tresp (Eds.).
25. Dayan P., Kakade S., Montague P.R. Learning and selective attention. Nat. Neurosci. 2000, 3:1218-1223.
26. Delgado M.R., Locke H.M., Stenger V.A., Fiez J.A. Dorsal striatum responses to reward and punishment: effects of valence and magnitude manipulations. Cogn. Affect. Behav. Neurosci. 2003, 3:27-38.
27. Doya K. Reinforcement learning in continuous time and space. Neural Computation 2000, 12:219-245.
28. Doya K. Metalearning and neuromodulation. Neural Networks 2002, 15:495-506.
29. Fiorillo C.D., Tobler P.N., Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 2003, 299:1898-1902.
30. Floresco S.B., West A.R., Ash B., et al. Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nat. Neurosci. 2003, 6:968-973.
31. Franklin K.B.J., McCoy S.N. Pimozide-induced extinction in rats: stimulus control of responding rules out motor deficit. Pharmacol. Biochem. Behav. 1979, 11:71-75.
32. Geisler S., Zahm D.S. Afferents of the ventral tegmental area in the rat-anatomical substratum for integrative functions. J. Comp. Neurol. 2005, 490:270-294.
33. Goto Y., Grace A.A. Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat. Neurosci. 2005, 8:805-812.
34. Grace A.A. Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 1991, 41:1-24.
35. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 2006, 26:8360-8367.
36. Hollerman J.R., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1998, 1:304-309.
37. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 2000, 96:651-656.
38. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia 1995, 249-270. MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
39. Howard R.A. Dynamic Programming and Markov Processes 1960, MIT Press, Cambridge, MA.
40. Ikemoto S., Panksepp J. The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res. Rev. 1999, 31:6-41.
41. Jensen J., Smith A.J., Willeit M., et al. Separate brain regions code for salience vs valence during reward prediction in humans. Hum. Brain Mapp. 2007, 28:294-302.
42. Joel D., Weiner I. Striatal contention scheduling and the split circuit scheme of basal ganglia-thalamocortical circuitry: from anatomy to behaviour. Conceptual Advances in Brain Research: Brain Dynamics and the Striatal Complex 1999, 209-236. Harwood Academic Publishers, New York, NY. R. Miller, J. Wickens (Eds.).
43. Joel D., Niv Y., Ruppin E. Actor-Critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 2002, 15:535-547.
44. Kacelnik A. Normative and descriptive models of decision making: time discounting and risk sensitivity. Characterizing Human Psychological Adaptations: Ciba Foundation Symposium 208 1997, 51-70. Wiley, Chichester. G.R. Bock, G. Cardew (Eds.).
45. Kakade S., Dayan P. Dopamine: generalization and bonuses. Neural Networks 2002, 15:549-559.
46. Kamin L.J. Predictability, surprise, attention, and conditioning. Punishment and Aversive Behavior 1969, 242-259. Appleton Century Crofts, New York, NY. B.A. Campbell, R.M. Church (Eds.).
47. Kehoe, E.J. (1977). Effects of serial compound stimuli on stimulus selection in classical conditioning of the rabbit nictitating membrane response. PhD thesis, university of Iowa.
48. Knutson B., Gibbs S.E.B. Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacol (Berl.), 2007, 191:813-822.
49. Knutson B., Adams C.M., Fong G.W., Hommer D. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J. Neurosci. 2001, 21:RC159.
50. Knutson B., Fong G.W., Adams C.M., et al. Dissociation of reward anticipation and outcome with event-related fmri. NeuroReport 2001, 12:3683-3687.
51. Knutson B., Fong G.W., Bennett S.M., et al. A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fmri. NeuroImage 2003, 18:263-272.
52. Kobayashi Y., Okada K.-I. Reward prediction error computation in the pedunculo-pontine tegmental nucleus neurons. Ann. N.Y. Acad. Sci. 2007, 1104:310-323.
53. Konda V.R., Tsitsiklis J.N. On Actor-Critic algorithms. SIAM J. Control Optimization, 2003, 42:1143-1166.
54. Konorski J. Conditioned Reflexes and Neuron Organization 1948, Cambridge University Press, New York, NY.
55. Kremer E.F. The Rescorla-Wagner model: losses in associative strength in compound conditioned stimuli. J. Exp. Psychol. Animal Behav. Proc. 1978, 4:22-36.
56. Lewicki M.S., Olshausen B.A. A probabilistic framework for the adaptation and comparison of image codes. J. Opt. Soc. Am. A 1999, 16:1587-1601.
57. Li J., McClure S.M., King-Casas B., Montague P.R. Policy adjustment in a dynamic economic game. PLoS ONE 2006, 1:e103.
58. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopaminergic neurons during learning of behavioral reactions. J. Neurophysiol. 1992, 67:145-163.
59. Logothetis N.K. The underpinnings of the BOLD functional magnetic resonance imaging signal. J. Neurosci. 2003, 23:3963-3971.
60. Lohrenz T., McCabe K., Camerer C.F., Montague P.R. Neural signature of fictive learning signals in a sequential investment task. Proc. Nat. Acad. Sci. USA 2007, 104:9493-9498.
61. Marr D. Vision: A Computational Approach 1982, Freeman & Co, San Francisco, CA.
62. Matsumoto M., Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 2007, 447:1111-1115.
63. McClure S.M., Daw N.D., Montague P.R. A computational substrate for incentive salience. Trends Neurosci. 2003, 26:423-428.
64. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
65. McClure S.M., Li J., Tomlin D., et al. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 2004, 44:379-387.
66. Menon M., Jensen J., Vitcu I., et al. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: effects of dopaminergic modulation. Biol. Psych. 2007, 62:765-772.
67. Miller R., Wickens J.R. Corticostriatal cell assemblies in selective attention and in representation of predictable and controllable events. Concepts Neurosci. 1991, 2:65-95.
68. Mirenowicz J., Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 1996, 379:449-451.
69. Montague P.R., Dayan P., Nowlan S.J., et al. Using aperiodic reinforcement for directed self-organization. Advances in Neural Information Processing Systems 1993, Vol. 5:969-976. Morgan Kaufmann, San Mateo, CA. C.L. Giles, S.J. Hanson, J.D. Cowan (Eds.).
70. Montague P.R., Dayan P., Sejnowski T.J. Foraging in an uncertain environments using predictive hebbian learning. Advances in Neural Information Processing Systems 1994, Vol. 6:598-605. Morgan Kaufmann, San Mateo, CA. T. Tesauro, J.D. Cowan (Eds.).
71. Montague P.R., Dayan P., Person C., Sejnowski T.J. Bee foraging in uncertain environments using predictive Hebbian learning. Nature 1995, 377:725-728.
72. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive hebbian learning. J. Neurosci. 1996, 16:1936-1947.
73. Montague P.R., McClure S.M., Baldwin P.R., et al. Dynamic gain control of dopamine delivery in freely moving animals. J. Neurosci. 2004, 24:1754-1759.
74. Morris G., Arkadir D., Nevet A. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 2004, 43:133-143.
75. Morris G., Nevet A., Arkadir D., et al. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
76. Ng, A.Y., Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In: Proceedings of the Sixteenth International Conference on Machine Learning. San Francisco, CA: Morgan Kaufmann, pp. 278-287.
77. Nicola S.M., Surmeier J., Malenka R.C. Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu. Rev. Neurosci. 2000, 23:185-215.
78. Niv Y. Cost, benefit, tonic, phasic: what do response rates tell us about dopamine and motivation?. Ann. NY Acad. Sci. 2007, 1104:357-376.
79. Niv, Y. (2007b). The Effects of Motivation on Habitual Instrumental Behavior. Unpublished doctoral dissertation, The Hebrew University of Jerusalem.
80. Niv Y., Daw N.D., Dayan P. How fast to work: response vigor, motivation and tonic dopamine. Advances in Neural Information Processing Systems 2005, Vol. 18:1019-1026. MIT Press, Cambridge, MA. Y. Weiss, B. Schölkopf, J. Platt (Eds.).
81. Niv Y., Duff M.O., Dayan P. Dopamine, uncertainty and TD learning. Behav. Brain Func. 2005, 1:6.
82. Niv Y., Joel D., Dayan P. A normative perspective on motivation. Trends Cogn. Science, 2006, 10:375-381.
83. Niv Y., Daw N.D., Dayan P. Choice values. Nat. Neurosci. 2006, 9:987-988.
84. Niv Y., Daw N.D., Joel D., Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacol. (Berl.), 2007, 191:507-520.
85. Nomoto K., Watanabe T., Sakagami M. Dopamine responses to complex reward-predicting stimuli. Soc. Neurosci. Abst. 2007, 33:749.5.
86. O'Doherty J.P., Deichmann R., Critchley H.D., Dolan R.J. Neural responses during anticipation of a primary taste reward. Neuron 2002, 33:815-826.
87. O'Doherty J., Dayan P., Friston K., et al. Temporal difference learning model accounts for responses in human ventral striatum and orbitofrontal cortex during Pavlovian appetitive learning. Neuron 2003, 38:329-337.
88. O'Doherty J.P., Dayan P., Schultz J., et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454.
89. Pagnoni G., Zink C.F., Montague P.R., Berns G.S. Activity in human ventral striatum locked to errors of reward prediction. Nat. Neurosci. 2002, 5:97-98.
90. Pessiglione M., Seymour B., Flandin G., et al. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 2006, 442:1042-1045.
91. Preuschoff K., Bossaerts P., Quartz S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 2006, 51:381-390.
92. Redgrave P., Gurney K. The short-latency dopamine signal: a role in discovering novel actions?. Nat. Rev. Neurosci. 2006, 7:967-975.
93. Redgrave P., Prescott T.J., Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends Neurosci. 1999, 22:146-151.
94. Rescorla R.A. Reduction in effectiveness of reinforcement after prior excitatory conditioning. Learning Motiv. 1970, 1:372-381.
95. Rescorla R.A., Lolordo V.M. Inhibition of avoidance behavior. J. Comp. Physiol. Psychol. 1968, 59:406-412.
96. Rescorla R.A., Wagner A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory 1972, 64-99. Appleton Century Crofts, New York, NY. A.H. Black, W.F. Prokasy (Eds.).
97. Reynolds G.S. Attention in the pigeon. J. Exp. Anal. Behav. 1961, 4:203-208.
98. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neurosci. 2007, 10:1615-1624.
99. Romo R., Schultz W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-intiated arm movements. J. Neurophysiol. 1990, 63:592-606.
100. Salamone J.D., Correa M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 2002, 137:3-25.
101. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310:1337-1340.
102. Samuel A.L. Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 1959, 3:210-229.
103. Schönberg T., Daw N.D., Joel D., O'Doherty J.P. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J. Neurosci. 2007, 27:12860-12867.
104. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 1998, 80:1-27.
105. Schultz W. Getting formal with dopamine and reward. Neuron 2002, 36:241-263.
106. Schultz W., Apicella P., Scarnati E., Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci. 1992, 12:4595-4610.
107. Schultz W., Apicella P., Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during succesive steps of learning a delayed response task. J. Neurosci. 1993, 13:900-913.
108. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
109. Schwartz, A. (1993). Thinking locally to act globally: a novel approach to reinforcement learning. In: Proceedings of the Fifth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Lawrence Erlbaum Associates, pp. 906-911.
110. Seymour B., O'Doherty J.P., Dayan P., et al. Temporal difference models describe higher order learning in humans. Nature 2004, 429:664-667.
111. Sutton, R.S. (1978). A Unified Theory of Expectation in Classical and Instrumental Conditioning. Unpublished Bsc thesis, Stanford University.
112. Sutton R.S. Learning to predict by the method of temporal difference. Machine Learning 1988, 3:9-44.
113. Sutton R.S., Barto A.G. Time-derivative models of Pavlovian reinforcement. Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990, 497-537. MIT Press, Cambridge, MA. M. Gabriel, J. Moore (Eds.).
114. Sutton R.S., Barto A.G. Reinforcement Learning: An Introduction 1998, MIT Press, Cambridge, MA.
115. Takikawa Y., Kawagoe R., Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. J. Neurophysiol. 2004, 92:2520-2529.
116. Tobler P.N., Dickinson A., Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 2003, 23:10402-10410.
117. Tobler P.N., Fiorillo C.D., Schultz W. Adaptive coding of reward value by dopamine neurons. Science 2005, 307:1642-1645.
118. Tom S.M., Fox C.R., Trepel C., Poldrack R.A. The neural basis of loss aversion in decision-making under risk. Science 2007, 315:515-518.
119. Ungless M.A., Magill P.J., Bolam J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 2004, 303:2040-2042.
120. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 2001, 412:43-48.
121. Watkins, C.J.C.H. (1989). Learning with Delayed Rewards. Unpublished doctoral dissertation, Cambridge University, Cambridge.
122. Weiner I., Joel D. Dopamine in schizophrenia: dysfunctional information processing in basal ganglia-thalamocortical split circuits. Handbook of Experimental Pharmacology Vol. 154/II, Dopamine in the CNS II 2002, 417-472. Springer Verlag, Berlin. G.D. Chiara (Ed.).
123. Werbos P.J. Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook 1977, 22:25-38.
124. Wickens J., Kötter R. Cellular models of reinforcement. Models of Information Processing in the Basal Ganglia 1995, 187-214. MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
125. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 1992, 8:229-256.
126. Willner P., Towell A., Muscat R. Effects of amphetamine and pimozide on reinforcement and motor parameters in variable-interval performance. J. Psychopharmacol. 1987, 1:140-153.
127. Wise R.A. Neuroleptics and operant behavior: the anhedonia hypothesis. Behav. Brain Sci. 1982, 5:39-53.
128. Wise R.A. Dopamine, learning and motivation. Nat. Rev. Neurosci. 2004, 5:483-495.
129. Wise R.A., Spindler J., de Wit H., Gerberg G.J. Neuroleptic-induced "anhedonia" in rats: pimozide blocks reward quality of food. Science 1978, 201:262-264.
130. Wise R.A., Spindler J., Legault L. Major attenuation of food reward with performance-sparing doses of pimozide in the rat. Can. J. Psychol. 1978, 32:77-85.
131. Yu A.J., Dayan P. Acetylcholine in cortical inference. Neural Networks 2002, 15:719-730.
132. Yu A.J., Dayan P. Uncertainty, neuromodulation, and attention. Neuron 2005, 46:681-692.
133. Zald D.H., Boileau I., El-Dearedy W., et al. Dopamine transmission in the human striatum during monetary reward tasks. J. Neurosci. 2004, 24:4105-4112.