Reinforcement learning in the brain

Journal of Mathematical Psychology

Volumn 53, Issue 3, 2009, Pages 139-154

  Niv, Yael ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 67349283062     PISSN: 00222496     EISSN: 10960880     Source Type: Journal    
DOI: 10.1016/j.jmp.2008.12.005     Document Type: Article

References (174)

1. Adams C.D. Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology 34B (1982) 77-98
2. Adams C.D., and Dickinson A. Instrumental responding following reinforcer devaluation. Quarterly Journal of Experimental Psychology 33B (1981) 109-121
3. Arbuthnott G.W., Ingham C.A., and Wickens J.R. Dopamine and synaptic plasticity in the neostriatum. Journal of Anatomy 196 Pt 4 (2000) 587-596
4. Attwell D., and Iadecola C. The neural basis of functional brain imaging signals. Trends in Neuroscience 25 12 (2002) 621-625
5. Baird, L. C. (1993). Advantage updating. Tech. rep. no. WL-TR-93-1146. Dayton, OH: Wright-Patterson Air Force Base
6. Baird L.C. Residual algorithms: Reinforcement learning with function approximation. In: Prieditis A., and Russell S. (Eds). Proceedings of the 12th international conference on machine learning (1995), Morgan Kaufman, San Mateo, CA 30-37
7. Balleine B.W. Neural bases of food-seeking: Affect, arousal and reward in corticostriatolimbic circuits. Physiology and Behaviour 86 5 (2005) 717-730
8. Balleine B.W., and Dickinson A. Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology 37 4-5 (1998) 407-419
9. Balleine B.W., Garner C., Gonzalez F., and Dickinson A. Motivational control of heterogeneous instrumental chains. Journal of Experimental Psychology: Animal Behavior Processes 21 3 (1995) 203-217
10. Barto A.G. Reinforcement learning control. Current Opinion in Neurobiology 4 6 (1994) 888-893
11. Barto A.G. Adaptive critic and the basal ganglia. In: Houk J.C., Davis J.L., and Beiser D.G. (Eds). Models of information processing in the basal ganglia (1995), MIT Press, Cambridge 215-232
12. Barto A.G., and Mahadevan S. Recent advances in hierarchical reinforcement learning. Discrete Event Systems Journal 13 (2003) 44-77
13. Barto A.G., Sutton R.S., and Anderson C.W. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man and Cybernetics 13 (1983) 834-846
14. Barto A.G., Sutton R.S., and Watkins C.J.C.H. Sequential decision problems and neural networks. In: Touretzky D.S. (Ed). Advances in neural information processing systems 2 (1989), MIT Press, Cambridge, MA 686-693
15. Barto A.G., Sutton R.S., and Watkins C.J.C.H. Learning and sequential decision making. In: Gabriel M., and Moore J. (Eds). Learning and computational neuroscience: Foundations of adaptive networks (1990), MIT Press, Cambridge, MA 539-602
16. Bayer H.M., and Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47 1 (2005) 129-141
17. Bayer H.M., Lau B., and Glimcher P.W. Statistics of midbrain dopamine neuron spike trains in the awake primate. Journal of Neurophysiology 98 3 (2007) 1428-1439
18. Behrens T.E.J., Woolrich M.W., Walton M.E., and Rushworth M.F.S. Learning the value of information in an uncertain world. Nature Neuroscience 10 9 (2007) 1214-1221
19. Bellman R.E. Dynamic programming (1957), Princeton University Press, Princeton, NJ
20. Bergstrom B.P., and Garris P.A. 'Passive stabilization' of striatal extracellular dopamine across the lesion spectrum encompassing the presymptomatic phase of Parkinson's disease: A voltammetric study in the 6-OHDA lesioned rat. Journal of Neurochemistry 87 5 (2003) 1224-1236
21. Berns G.S., McClure S.M., Pagnoni G., and Montague P.R. Predictability modulates human brain response to reward. Journal of Neuroscience 21 8 (2001) 2793-2798
22. Berridge K.C. Espresso reward learning, hold the dopamine: Theoretical comment on robinson et al. (2005). Behavioral Neuroscience 119 1 (2005) 336-341
23. Berridge K.C. The debate over dopamine's role in reward: The case for incentive salience. Psychopharmacology (Berl) 191 3 (2007) 391-431
24. Berridge K.C., and Robinson T.E. What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience?. Brain Research Review 28 (1998) 309-369
25. Bertsekas D.P., and Tsitsiklis J.N. Neuro-dynamic programming (1996), Athena Sc
26. Botvinick M.M. Hierarchical models of behavior and prefrontal function. Trends in Cognitive Sciences 12 5 (2008) 201-208
27. Botvinick M.M., Niv Y., and Barto A.C. Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition (2008)
28. Bush R.R., and Mosteller F. A mathematical model for simple learning. Psychological Review 58 (1951) 313-323
29. Cardinal R.N., Parkinson J.A., Hall J., and Everitt B.J. Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Biobehavioural Reviews 26 3 (2002) 321-352
30. Christoph G.R., Leonzio R.J., and Wilcox K.S. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. Journal of Neuroscience 6 3 (1986) 613-619
31. Costa R.M., Lin S.-C., Sotnikova T.D., Cyr M., Gainetdinov R.R., Caron M.G., et al. Rapid alterations in corticostriatal ensemble coordination during acute dopamine-dependent motor dysfunction. Neuron 52 2 (2006) 359-369
32. Daw N.D., Courville A.C., and Touretzky D.S. Timing and partial observability in the dopamine system. In: Dietterich T., Becker S., and Ghahramani Z. (Eds). Advances in neural information processing systems Vol. 14 (2002), MIT Press, Cambridge, MA
33. Daw N.D., Courville A.C., and Touretzky D.S. Representation and timing in theories of the dopamine system. Neural Computation 18 (2006) 1637-1677
34. Daw N.D., Kakade S., and Dayan P. Opponent interactions between serotonin and dopamine. Neural Networks 15 4-6 (2002) 603-616
35. Daw N.D., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8 12 (2005) 1704-1711
36. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., and Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 441 7095 (2006) 876-879
37. Daw N.D., and Touretzky D.S. Long-term reward prediction in TD models of the dopamine system. Neural Computation 14 11 (2002) 2567-2583
38. Day J.J., Roitman M.F., Wightman R.M., and Carelli R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nature Neuroscience 10 8 (2007) 1020-1028
39. Dayan P., and Abbott L.F. Theoretical neuroscience: Computational and mathematical modeling of neural systems (2001), MIT Press
40. Dayan P., Kakade S., and Montague P.R. Learning and selective attention. Nature Neuroscience 3 Suppl (2000) 1218-1223
41. Dayan P., and Niv Y. Reinforcement learning: The good, the bad and the ugly. Current Opinion in Neurobiology 18 2 (2008) 185-196
42. Dayan P., Niv Y., Seymour B., and Daw N.D. The misbehavior of value and the discipline of the will. Neural Networks 19 8 (2006) 1153-1160
43. Delgado M.R., Locke H.M., Stenger V.A., and Fiez J.A. Dorsal striatum responses to reward and punishment: Effects of valence and magnitude manipulations. Cognitive Affect Behavioural Neuroscience 3 1 (2003) 27-38
44. Dickinson A. Actions and habits: The development of behavioural autonomy. Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences 308 1135 (1985) 67-78
45. Dickinson A., and Balleine B.W. The role of learning in the operation of motivational systems. In: Gallistel C.R. (Ed). Learning, motivation and emotion Vol. 3 (2002), John Wiley & Sons, New York 497-533
46. Dickinson A., Nicholas D.J., and Adams C.D. The effect of instrumental training contingency on susceptibility to reinforcer devaluation. Quarterly Journal of Experimental Psychology 35B (1983) 35-51
47. Doya K. Reinforcement learning in continuous time and space. Neural Computation 12 1 (2000) 219-245
48. Fiorillo C.D., Tobler P.N., and Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299 5614 (2003) 1898-1902
49. Floresco S.B., West A.R., Ash B., Moore H., and Grace A.A. Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nature Neuroscience 6 9 (2003) 968-973
50. Gallistel C.R., and Gibbon J. Time, rate and conditioning. Psychological Review 107 (2000) 289-344
51. Geisler S., and Zahm D.S. Afferents of the ventral tegmental area in the rat-anatomical substratum for integrative functions. The Journal of Comparative Neurology 490 3 (2005) 270-294
52. Gibbon J. Scalar Expectancy Theory and Weber's law in animal timing. Psychological Review 84 3 (1977) 279-325
53. Goto Y., and Grace A.A. Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nature Neuroscience 8 (2005) 805-812
54. Grace A.A. Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: A hypothesis for the etiology of schizophrenia. Neuroscience 41 1 (1991) 1-24
55. Hampton A.N., Bossaerts P., and O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience 26 32 (2006) 8360-8367
56. Hampton A.N., and O'Doherty J.P. Decoding the neural substrates of reward-related decision making with functional mri. Proceedings of the National Academy of Sciences USA 104 4 (2007) 1377-1382
57. Holland P.C., and Gallagher M. Amygdala circuitry in attentional and representational processes. Trends in Cognitive Sciences 3 2 (1999) 65-73
58. Hollerman J.R., and Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1 (1998) 304-309
59. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96 4 (2000) 651-656
60. Houk J.C., Adams J.L., and Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk J.C., Davis J.L., and Beiser D.G. (Eds). Models of information processing in the basal ganglia (1995), MIT Press, Cambridge 249-270
61. Howard R.A. Dynamic Programming and Markov processes (1960), MIT Press
62. Ikemoto S., and Panksepp J. The role of nucleus accumbens dopamine in motivated behavior: A unifying interpretation with special reference to reward-seeking. Brain Research Reviews 31 (1999) 6-41
63. Jensen J., Smith A.J., Willeit M., Crawley A.P., Mikulis D.J., Vitcu I., and Kapur S. Separate brain regions code for salience vs. valence during reward prediction in humans. Human Brain Mapping 28 (2007) 294-302
64. Joel D., Niv Y., and Ruppin E. Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks 15 (2002) 535-547
65. Joel D., and Weiner I. The organization of the basal ganglia-thalamocortical cicuits: open interconnected rather than closed segregated. Neuroscience 63 (1994) 363-379
66. Joel D., and Weiner I. Striatal contention scheduling and the split circuit scheme of basal ganglia-thalamocortical circuitry: From anatomy to behaviour. In: Miller R., and Wickens J. (Eds). Conceptual advances in brain research: Brain dynamics and the striatal complex (1999), Harwood Academic Publishers 209-236
67. Kacelnik A. Normative and descriptive models of decision making: Time discounting and risk sensitivity. In: Bock G.R., and Cardew G. (Eds). Characterizing human psychological adaptations: Ciba Foundation symposium 208 (1997), Wiley, Chichester 51-70
68. Kakade S., and Dayan P. Dopamine: Generalization and bonuses. Neural Networks 15 4-6 (2002) 549-559
69. Kamin L.J. Predictability, surprise, attention, and conditioning. In: Campbell B.A., and Church R.M. (Eds). Punishment and aversive behavior (1969), Appleton-Century-Crofts, New York 242-259
70. Kehoe, E. J. (1977). Effects of serial compound stimuli on stimulus selection in classical conditioning of the rabbit nictitating membrane response. Unpublished doctoral dissertation, University of Iowa
71. Killcross S., and Blundell P. Associative representationsof emotionally significant outcomes. In: Moore S., and Oaksford M. (Eds). Emotional cognition. from brain to behaviour Vol. 44 (2002), John Benjamins Publishing Company, Amsterdam, Philadelphia 35-73
72. Killcross S., and Coutureau E. Coordination of actions and habits in the medial prefrontal conrtex of rats. Cerebral Cortex 13 (2003) 400-408
73. Knutson B., Adams C.M., Fong G.W., and Hommer D. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. Journal of Neuroscience 21 16 (2001) RC159
74. Knutson B., Delgado M.R., and Philips P.E.M. Representation of subjective value in the striatum. In: Glimcher P.W., Camerer C., Fehr E., and Poldrack R. (Eds). Neuroeconomics: Decision making and the brain (2008), Academic Press, New York, NY
75. Knutson B., Fong G.W., Adams C.M., Varner J.L., and Hommer D. Dissociation of reward anticipation and outcome with event-related fMRI. Neuroreport 12 17 (2001) 3683-3687
76. Knutson B., Fong G.W., Bennett S.M., Adams C.M., and Hommer D. A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: Characterization with rapid event-related fMRI. Neuroimage 18 2 (2003) 263-272
77. Knutson B., and Gibbs S.E.B. Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl) 191 3 (2007) 813-822
78. Kobayashi Y., and Okada K.-I. Reward prediction error computation in the pedunculopontine tegmental nucleus neurons. Annals of the New York Academy of Science 1104 (2007) 310-323
79. Konda V.R., and Tsitsiklis J.N. On actor-critic algorithms. SIAM Journal of Control Optimization 42 4 (2003) 1143-1166
80. Konorski J. Conditioned reflexes and neuron organization (1948), Cambridge University Press, New York
81. Kremer E.F. The Rescorla-Wagner model: Losses in associative strength in compound conditioned stimuli. Journal of Experimental Psychology: Animal Behavior Processes 4 1 (1978) 22-36
82. Krimer L.S., Muly E.C., Williams G.V., and Goldman-Rakic P.S. Dopaminergic regulation of cerebral cortical microcirculation. Nature Neuroscience 1 4 (1998) 286-289
83. Lecourtier L., Defrancesco A., and Moghaddam B. Differential tonic influence of lateral habenula on prefrontal cortex and nucleus accumbens dopamine release. European Journal of Neuroscience 27 7 (2008) 1755-1762
84. Li J., McClure S.M., King-Casas B., and Montague P.R. Policy adjustment in a dynamic economic game. PLoS ONE 1 1 (2006) e103
85. Ljungberg T., Apicella P., and Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology 67 1 (1992) 145-163
86. Lobo M.K., Cui Y., Ostlund S.B., Balleine B.W., and Yang X.W. Genetic control of instrumental conditioning by striatopallidal neuron-specific S1P receptor Gpr6. Nature Neuroscience 10 11 (2007) 1395-1397
87. Logothetis N.K. The underpinnings of the BOLD functional magnetic resonance imaging signal. Journal of Neuroscience 23 10 (2003) 3963-3971
88. Logothetis N.K., and Wandell B.A. Interpreting the bold signal. Annual Review of Physiology 66 (2004) 735-769
89. Marr D. Vision: A computational approach (1982), Freeman & Co, San Francisco
90. Matsumoto M., and Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447 (2007) 1111-1115
91. McClure S.M., Berns G.S., and Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 38 2 (2003) 339-346
92. McClure S.M., Daw N.D., and Montague P.R. A computational substrate for incentive salience. Trends in Neuroscience 26 8 (2003) 423-428
93. McClure S.M., Li J., Tomlin D., Cypert K.S., Montague L.M., and Montague P.R. Neural correlates of behavioral preference for culturally familiar drinks. Neuron 44 2 (2004) 379-387
94. Menon M., Jensen J., Vitcu I., Graff-Guerrero A., Crawley A., Smith M.A., et al. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: Effects of dopaminergic modulation. Biological Psychiatry 62 7 (2007) 765-772
95. Miller R., and Wickens J.R. Corticostriatal cell assemblies in selective attention and in representation of predictable and controllable events. Concepts in Neuroscience 2 1 (1991) 65-95
96. Mirenowicz J., and Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379 (1996) 449-451
97. Montague P.R., Dayan P., Nowlan S.J., Pouget A., and Sejnowski T.J. Using aperiodic reinforcement for directed self-organization. In: Giles C.L., Hanson S.J., and Cowan J.D. (Eds). Advances in neural information processing systems Vol. 5 (1993), Morgan Kaufmann, San Mateo, CA 969-976
98. Montague P.R., Dayan P., Person C., and Sejnowski T.J. Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377 (1995) 725-728
99. Montague P.R., Dayan P., and Sejnowski T.J. Foraging in an uncertain environments using predictive hebbian learning. In: Tesauro, and Cowan J.D. (Eds). Advances in neural information processing systems Vol. 6 (1994) 598-605
100. Montague P.R., Dayan P., and Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal of Neuroscience 16 5 (1996) 1936-1947
101. Montague P.R., McClure S.M., Baldwin P.R., Phillips P.E.M., Budygin E.A., Stuber G.D., et al. Dynamic gain control of dopamine delivery in freely moving animals. Journal of Neuroscience 24 7 (2004) 1754-1759
102. Morris G., Arkadir D., Nevet A., Vaadia E., and Bergman H. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43 1 (2004) 133-143
103. Morris G., Nevet A., Arkadir D., Vaadia E., and Bergman H. Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience 9 8 (2006) 1057-1063
104. Nakamura K., Matsumoto M., and Hikosaka O. Reward-dependent modulation of neuronal activity in the primate dorsal raphe nucleus. Journal of Neuroscience 28 20 (2008) 5331-5343
105. Nicola S.M., Surmeier J., and Malenka R.C. Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annual Reviews in Neuroscience 23 (2000) 185-215
106. Niv, Y. (2007). The effects of motivation on habitual instrumental behavior. Unpublished doctoral dissertation, The Hebrew University of Jerusalem
107. Niv Y., Daw N.D., and Dayan P. How fast to work: Response vigor, motivation and tonic dopamine. In: Weiss Y., Schölkopf B., and Platt J. (Eds). Advances in neural information processing systems: Vol. 18 (2005), MIT Press 1019-1026
108. Niv Y., Daw N.D., and Dayan P. Choice values. Nature Neuroscience 9 8 (2006) 987-988
109. Niv Y., Daw N.D., Joel D., and Dayan P. Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191 3 (2007) 507-520
110. Niv Y., Duff M.O., and Dayan P. Dopamine, uncertainty and TD learning. Behavioral and Brain Functions 1 (2005) 6
111. Niv Y., Joel D., and Dayan P. A normative perspective on motivation. Trends in Cognitive Sciences 10 8 (2006) 375-381
112. Niv Y., and Schoenbaum G. Dialogues on prediction errors. Trends in Cognitive Sciences 12 7 (2008) 265-272
113. Nomoto K., Watanabe T., and Sakagami M. Dopamine responses to complex reward-predicting stimuli. Society for Neuroscience abstracts Vol. 33 (2007) 749.5
114. O'Doherty J., Dayan P., Friston K., Critchley H., and Dolan R. Temporal difference learning model accounts for responses in human ventral striatum and orbitofrontal cortex during Pavlovian appetitive learning. Neuron 38 (2003) 329-337
115. O'Doherty J.P., Dayan P., Schultz J., Deichmann R., Friston K., and Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304 5669 (2004) 452-454
116. O'Doherty J.P., Deichmann R., Critchley H.D., and Dolan R.J. Neural responses during anticipation of a primary taste reward. Neuron 33 5 (2002) 815-826
117. Pagnoni G., Zink C.F., Montague P.R., and Berns G.S. Activity in human ventral striatum locked to errors of reward prediction. Nature Neuroscience 5 2 (2002) 97-98
118. Parent A., and Hazrati L.N. Anatomical aspects of information processing in primate basal ganglia. Trends in Neurosciences 16 3 (1993) 111-116
119. Parr R., and Russell S. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems: Vol. 10 (1998), MIT Press
120. Pessiglione M., Seymour B., Flandin G., Dolan R.J., and Frith C.D. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442 7106 (2006) 1042-1045
121. Preuschoff K., Bossaerts P., and Quartz S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51 3 (2006) 381-390
122. Redgrave P., and Gurney K. The short-latency dopamine signal: A role in discovering novel actions?. Nature Reviews Neuroscience 7 12 (2006) 967-975
123. Redgrave P., Prescott T.J., and Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends in Neurosciences 22 4 (1999) 146-151
124. Redish A.D., Jensen S., Johnson A., and Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review 114 3 (2007) 784-805
125. Rescorla R.A. Reduction in effectiveness of reinforcement after prior excitatory conditioning. Learning and Motivation 1 (1970) 372-381
126. Rescorla R.A., and Lolordo V.M. Inhibition of avoidance behavior. Journal of Comparative and Physiological Psychology 59 (1968) 406-412
127. Rescorla R.A., and Wagner A.R. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black A.H., and Prokasy W.F. (Eds). Classical conditioning II: Current research and theory (1972), Appleton-Century-Crofts, New York, NY 64-99
128. Reynolds G.S. Attention in the pigeon. Journal of the Experimental Analysis of Behavior 4 (1961) 203-208
129. Reynolds J.N., Hyland B.I., and Wickens J.R. A cellular mechanism of reward-related learning. Nature 413 6851 (2001) 67-70
130. Roesch M.R., Calu D.J., and Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience 10 12 (2007) 1615-1624
131. Romo R., and Schultz W. Dopamine neurons of the monkey midbrain: Contingencies of responses to active touch during self-intiated arm movements. The Journal of Neurophysiology 63 (1990) 592-606
132. Salamone J.D., and Correa M. Motivational views of reinforcement: Implications for understanding the behavioral functions of nucleus accumbens dopamine. Behavioural Brain Research 137 (2002) 3-25
133. Samejima K., Ueda Y., Doya K., and Kimura M. Representation of action-specific reward values in the striatum. Science 310 5752 (2005) 1337-1340
134. Samuel A.L. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development 3 (1959) 210-229
135. Schönberg T., Daw N.D., Joel D., and O'Doherty J.P. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. Journal of Neuroscience 27 47 (2007) 12860-12867
136. Schultz W. Predictive reward signal of dopamine neurons. Journal of Neurophysiology 80 (1998) 1-27
137. Schultz W. Getting formal with dopamine and reward. Neuron 36 2 (2002) 241-263
138. Schultz W., Apicella P., and Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience 13 3 (1993) 900-913
139. Schultz W., Apicella P., Scarnati E., and Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. Journal of Neuroscience 12 12 (1992) 4595-4610
140. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 275 (1997) 1593-1599
141. Schwartz, A. (1993). Thinking locally to act globally: A novel approach to reinforcement learning. In Proceedings of the fifteenth annual conference of the cognitive science society (pp. 906-911). Hillsdale, NJ: Lawrence Erlbaum Associates
142. Seymour B., O'Doherty J.P., Dayan P., Koltzenburg M., Jones A.K., Dolan R.J., et al. Temporal difference models describe higher-order learning in humans. Nature 429 6992 (2004) 664-667
143. Simsek, O., & Barto, A. G. (2004). Using relative novelty to identify useful temporal abstractions in reinforcement learning. In 21st international conference on machine learning
144. Skinner B.F. Two types of conditioned reflex and a pseudo type. Journal of General Psychology 12 (1935) 66-77
145. Stolle M., and Precup D. Learning options in reinforcement learning. Lecture Notes in Computer Science 2371 (2002) 212-223
146. Sutton, R. S. (1978). A unified theory of expectation in classical and instrumental conditioning. Unpublished bachelors thesis
147. Sutton R.S. Learning to predict by the method of temporal difference. Machine Learning 3 (1988) 9-44
148. Sutton R.S., and Barto A.G. Time-derivative models of Pavlovian reinforcement. In: Gabriel M., and Moore J. (Eds). Learning and computational neuroscience: Foundations of adaptive networks (1990), MIT Press 497-537
149. Sutton R.S., and Barto A.G. Reinforcement learning: An introduction (1998), MIT Press
150. Sutton R.S., Mcallester D., Singh S., and Mansour Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems 12 (2000), MIT Press 1057-1063
151. Sutton R.S., Precup D., and Singh S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112 (1999) 181-211
152. Takikawa Y., Kawagoe R., and Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. Journal of Neurophysiology 92 (2004) 2520-2529
153. Thorndike E.L. Animal intelligence: Experimental studies (1911), Macmillan, New York
154. Tobler P.N., Dickinson A., and Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. Journal of Neuroscience 23 32 (2003) 10402-10410
155. Tobler P.N., Fiorillo C.D., and Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307 5715 (2005) 1642-1645
156. Tom S.M., Fox C.R., Trepel C., and Poldrack R.A. The neural basis of loss aversion in decision-making under risk. Science 315 5811 (2007) 515-518
157. Ungless M.A., Magill P.J., and Bolam J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303 5666 (2004) 2040-2042
158. Valentin V.V., Dickinson A., and O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. Journal of Neuroscience 27 15 (2007) 4019-4026
159. Waelti P., Dickinson A., and Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412 (2001) 43-48
160. Walton M.E., Gan J.O., Barnes S.D., Evans S.B., and Phillips P.E.M. Subsecond dopamine release in cost-benefit decision making. Society for Neuroscience abstracts Vol. 32 (2006) 71.2
161. Watkins, C. J. C. H. (1989). Learning with delayed rewards. Unpublished doctoral dissertation, Cambridge University, Cambridge, UK
162. Weiner I., and Joel D. Dopamine in schizophrenia: Dysfunctional information processing in basal ganglia-thalamocortical split circuits. In: Chiara G.D. (Ed). Handbook of experimental pharmacology vol. 154/II, dopamine in the CNS II (2002), Springer-Verlag, Berlin 417-472
163. Werbos P.J. Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook 22 (1977) 25-38
164. Wickens J.R., Begg A.J., and Arbuthnott G.W. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high frequency stimulation of cortex in vitro. Neuroscience 70 1 (1996) 1-5
165. Wickens J.R., and Kötter R. Cellular models of reinforcement. In: Houk J.C., Davis J.L., and Beiser D.G. (Eds). Models of information processing in the basal ganglia (1995), MIT Press 187-214
166. Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8 3 (1992) 229-256
167. Wise R.A., Spindler J., de Wit H., and Gerberg G.J. Neuroleptic-induced "anhedonia" in rats: pimozide blocks reward quality of food. Science 201 4352 (1978) 262-264
168. Wise R.A., Spindler J., and Legault L. Major attenuation of food reward with performance-sparing doses of pimozide in the rat. Canadian Journal of Psychology 32 (1978) 77-85
169. Wittmann B.C., Daw N.D., Seymour B., and Dolan R.J. Striatal activity underlies novelty-based choice in humans. Neuron 58 6 (2008) 967-973
170. Yerkes R.M., and Morgulis S. The method of Pawlow in animal psychology. Psychological Bulletin 6 (1909) 257-273
171. Yin H.H., Knowlton B.J., and Balleine B.W. Lesions of the dosolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience 19 (2004) 181-189
172. Yin H.H., Knowlton B.J., and Balleine B.W. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. European Journal of Neuroscience 22 2 (2005) 505-512
173. Yin H.H., Ostlund S.B., Knowlton B.J., and Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience 22 2 (2005) 513-523
174. Zald D.H., Boileau I., El-Dearedy W., Gunn R., McGlone F., Dichter G.S., et al. Dopamine transmission in the human striatum during monetary reward tasks. Journal of Neuroscience 24 17 (2004) 4105-4112