Beyond simple reinforcement learning: The computational neurobiology of reward-learning and valuation

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 987-990

  O'Doherty, John P ^a  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Basal ganglia; Computational neuroscience; Conditioning; Decision making; Prefrontal cortex

Indexed keywords

DOPAMINE;

ARTICLE; BAYES THEOREM; COGNITION; CORPUS STRIATUM; DOPAMINERGIC SYSTEM; DORSAL STRIATUM; HIPPOCAMPUS; HUMAN; LEARNING; NONHUMAN; ORBITAL CORTEX; PREFRONTAL CORTEX; PRIORITY JOURNAL; REINFORCEMENT; REWARD; REWARD LEARNING; STIMULUS GENERALIZATION; STIMULUS RESPONSE; TEMPORAL DIFFERENCE REINFORCEMENT LEARNING; VENTROMEDIAL PREFRONTAL CORTEX;

COMPUTATIONAL BIOLOGY; DOPAMINE; HUMANS; LEARNING; NEUROBIOLOGY; REINFORCEMENT (PSYCHOLOGY); REWARD; TIME FACTORS;

EID: 84859311497     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2012.08074.x     Document Type: Article

References (43)

1. Aggarwal, M., Hyland, B.I. & Wickens, J.R. (2012) Neural control of dopamine neurotransmission: implications for reinforcement learning. Euro. J. Neurosci., 35, 1115-1123.
2. Balleine, B.W. & Dickinson, A. (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology, 37, 407-419.
3. Balleine, B.W. & O'Doherty, J.P. (2010) Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology, 35, 48-69.
4. Bayer, H.M. & Glimcher, P.W. (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47, 129-141.
5. Behrens, T.E., Woolrich, M.W., Walton, M.E. & Rushworth, M.F. (2007) Learning the value of information in an uncertain world. Nat. Neurosci., 10, 1214-1221.
6. Berridge, K.C. (2007) The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology, 191, 391-431.
7. Berridge, K.C. (2012) From prediction error to incentive salience: mesolimbic computation of reward motivation. Euro. J. Neurosci., 35, 1124-1143.
8. Berridge, K.C. & Robinson, T.E. (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Brain Res. Rev., 28, 309-369.
9. Bornstein, A.M. & Daw, N. (2012) Dissociating hippocampal and striatal contributions to sequential prediction learning. Euro. J. Neurosci., 35, 1011-1023.
10. Botvinick, M.M., Niv, Y. & Barto, A.C. (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition, 113, 262-280.
11. Collins, A. & Frank, M.J. (2012) How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Euro. J. Neurosci., 35, 1024-1035.
12. Daw, N.D. & Touretzky, D.S. (2002) Long-term reward prediction in TD models of the dopamine system. Neural Comput., 14, 2567-2583.
13. Daw, N.D., Niv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci., 8, 1704-1711.
14. Dayan, P. (2012) Instrumental vigour in punishment and reward. Euro. J. Neurosci., 35, 1152-1167.
15. Dezfouli, A. & Balleine, B.W. (2012) Habits, action sequences, and reinforcement learning. Euro. J. Neurosci., 35, 1036-1051.
16. Doya, K., Samejima, K., Katagiri, K. & Kawato, M. (2002) Multiple model-based reinforcement learning. Neural Comput., 14, 1347-1369.
17. Flagel, S.B., Clark, J.J., Robinson, T.E., Mayo, L., Czuj, A., Willuhn, I., Akers, C.A., Clinton, S.M., Phillips, P.E. & Akil, H. (2011) A selective role for dopamine in stimulus-reward learning. Nature, 469, 53-57.
18. Funamizu, A., Ito, M., Doya, K., Kanzaki, R. & Takahashi, H. (2012) Uncertainty in action-value estimation affects both action choice and learning rate of the choice behaviors of rats. Euro. J. Neurosci., 35, 1179-1188.
19. Gottfried, J.A., O'Doherty, J. & Dolan, R.J. (2003) Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science, 301, 1104-1107.
20. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci., 26, 8360-8367.
21. Hilario, M., Holloway, T., Jin, X. & Costa, R. (2012) Different dorsal striatum circuits mediate action discrimination and action generalization. Euro. J. Neurosci., 35, 1105-1114.
22. Johnson, A., van der Meer, M.A. & Redish, A.D. (2007) Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol., 17, 692-697.
23. Koechlin, E., Ody, C. & Kouneiher, F. (2003) The architecture of cognitive control in the human prefrontal cortex. Science, 302, 1181-1185.
24. Kurth-Nelson, Z., Bickel, W.K. & Redish, A.D. (2012) A Theoretical account of cognitive effects in delay discounting. Euro. J. Neurosci., 35, 1052-1064.
25. McDannald, M.A., Takahashi, Y., Lopatina, N., Pietras, B., Jones, J.L. & Schoenbaum, G. (2012) Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Euro. J. Neurosci., 35, 991-996.
26. Montague, P.R., Dayan, P. & Sejnowski, T.J. (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci., 16, 1936-1947.
27. Niv, Y., Daw, N.D., Joel, D. & Dayan, P. (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl), 191, 507-520.
28. Noonan, M.P., Kolling, N., Walton, M. & Rushworth, M. (2012) Re-evaluating the role of orbitofrontal cortex in reward and reinforcement. Euro. J. Neurosci., 35, 997-1010.
29. O'Reilly, J.X., Jbabdi, S. & Behrens, T.E. (2012) How can a Bayesian approach inform neuroscience? Euro. J. Neurosci., 35, 1168-1178.
30. Pan, X. & Sakagami, M. (2012) Category representation and generalization in the prefrontal cortex. Euro. J. Neurosci., 35, 1083-1091.
31. Paton, J.J., Belova, M.A., Morrison, S.E. & Salzman, C.D. (2006) The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature, 439, 865-870.
32. Pearce, J.M. & Hall, G. (1980) A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev., 87, 532-552.
33. Robinson, S., Sandstrom, S.M., Denenberg, V.H. & Palmiter, R.D. (2005) Distinguishing whether dopamine regulates liking, wanting, and/or learning about rewards. Behav. Neurosci., 119, 5-15.
34. Roesch, M., Esber, G.R., Li, J., Daw, N. & Schoenbaum, G. (2012) Surprise! Neural correlates of Pearce-Hall and Rescorla-Wagner coexist within the brain. Euro. J. Neurosci., 35, 1189-1199.
35. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. (2005) Representation of action-specific reward values in the striatum. Science, 310, 1337-1340.
36. Schoenbaum, G., Chiba, A.A. & Gallagher, M. (1998) Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci., 1, 155-159.
37. Schultz, W. (1998) Predictive reward signal of dopamine neurons. J. Neurophysiol., 80, 1-27.
38. Schultz, W., Dayan, P. & Montague, P.R. (1997) A neural substrate of prediction and reward. Science, 275, 1593-1599.
39. Seamans, J.K. & Yang, C.R. (2004) The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog. Neurobiol., 74, 1-58.
40. Smittenaar, P., Chase, H.W., Aarts, E., Nusselein, B., Bloem, B.R. & Cools, R. (2012) Decomposing effects of dopaminergic medication in Parkinson's disease on probabilistic action selection: learning or performance? Euro. J. Neurosci., 35, 1144-1151.
41. Sutton, R.S. & Barto, A.G. (1998) Reinforcement Learning. MIT Press, Cambridge, MA.
42. Venkatraman, V. & Huettel, S. (2012) Strategic control in decision making under uncertainty. Euro. J. Neurosci., 35, 1075-1082.
43. Wimmer, G.E., Daw, N.D. & Shohamy, D. (2012) Generalization of value in reinforcement learning by humans. Euro. J. Neurosci., 35, 1092-1104.