The structure of reinforcement-learning mechanisms in the human brain

Current Opinion in Behavioral Sciences

Volumn 1, Issue , 2015, Pages 94-100

  O'Doherty, John P ^a   Lee, Sang Wan ^a   McNamee, Daniel ^a  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BRAIN FUNCTION; CONCEPTUAL FRAMEWORK; DOPAMINERGIC NERVE CELL; HIPPOCAMPUS; HUMAN; LEARNING ALGORITHM; LEARNING THEORY; NEUROIMAGING; POSTERIOR PARIETAL CORTEX; PREDICTION; PREFRONTAL CORTEX; PROBABILITY; REINFORCEMENT LEARNING THEORY; REVIEW;

EID: 84920069670     PISSN: None     EISSN: 23521546     Source Type: Journal    
DOI: 10.1016/j.cobeha.2014.10.004     Document Type: Review

References (70)

1. Sutton R.S., Barto A.G. Reinforcement learning 1998, MIT Press, Cambridge, MA.
2. Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol 1998, 80:1-27.
3. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 1996, 16:1936-1947.
4. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
5. Schonberg T., Daw N.D., Joel D., O'Doherty J.P. Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 2007, 27:12860-12867.
6. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
7. Niv Y., Edlund J.A., Dayan P., O'Doherty J.P. Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. J Neurosci 2012, 32:551-562.
8. D'Ardenne K., McClure S.M., Nystrom L.E., Cohen J.D. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 2008, 319:1264-1267.
9. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 2006, 26:8360-8367.
10. Glascher J., Hampton A.N., O'Doherty J.P. Determining a role for ventromedial prefrontal cortex in encoding action-based value signals during reward-related decision making. Cereb Cortex 2009, 19:483-495.
11. Boorman E.D., Behrens T.E., Woolrich M.W., Rushworth M.F. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 2009, 62:733-743.
12. Kuvayev L., Sutton R.S. Model-based reinforcement learning with an approximate,learned model. Proceedings of the Ninth Yale Workshop on Adaptive and Learning Systems 1996.
13. Doya K., Samejima K., Katagiri K., Kawato M. Multiple model-based reinforcement learning. Neural Comput 2002, 14:1347-1369.
14. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
15. Dickinson A. Actions and habits: the development of a behavioural autonomy. Philos Trans R Soc Lond B Biol Sci 1985, 308:67-78.
16. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
17. Beierholm U.R., Anen C., Quartz S., Bossaerts P. Separate encoding of model-based and model-free valuations in the human brain. Neuroimage 2011, 58:955-962.
18. Lee S.W., Shimojo S., O'Doherty J.P. Neural computations underlying arbitration between model-based and model-free learning. Neuron 2014, 81:687-699.
19. Glascher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
20. Liljeholm M., Wang S., Zhang J., O'Doherty J.P. Neural correlates of the divergence of instrumental probability distributions. J Neurosci 2013, 33:12519-12527.
21. Simon D.A., Daw N.D. Neural correlates of forward planning in a spatial decision task in humans. J Neurosci 2011, 31:5526-5539.
22. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 2012, 15:786-791.
23. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 2004, 19:181-189.
24. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur J Neurosci 2009, 29:2225-2232.
25. Doll B.B., Simon D.A., Daw N.D. The ubiquity of model-based reinforcement learning. Curr Opin Neurobiol 2012, 22:1075-1081.
26. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
27. Gershman S.J., Niv Y. Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 2010, 20:251-256.
28. Ahmad S., Yu A.J. Active sensing as Bayes-optimal sequential decision-making. Proceedings of the Twenty-ninth Annual Conference on Uncertainty in Artificial Intelligence (UAI) 2013, 12-21.
29. Kwok C., Fox D. Reinforcement learning for sensing strategies. 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), vol. 4 2004, 3158-3163. Edited by: IEEE.
30. Poggio T., Ullman S. Vision: are models of object recognition catching up with the brain?. Ann N Y Acad Sci 2013, 1305:72-82.
31. Shadlen M.N., Kiani R. Decision making as a window on cognition. Neuron 2013, 80:791-806.
32. Wolpert D.M., Diedrichsen J., Flanagan J.R. Principles of sensorimotor learning. Nat Rev Neurosci 2011, 12:739-751.
33. Ridderinkhof K.R. Neurocognitive mechanisms of perception-action coordination: a review and theoretical integration. Neurosci Biobehav Rev 2014.
34. Robbins T.W. Dissociating executive functions of the prefrontal cortex. Philos Trans R Soc Lond B Biol Sci 1996, 351:1463-1470. discussion 1470-1461.
35. Milner B. Effects of different brain lesions on card sorting: the role of the frontal lobes. Arch Neurol 1963, 9:90-100.
36. Wunderlich K., Beierholm U.R., Bossaerts P., O'Doherty J.P. The human prefrontal cortex mediates integration of potential causes behind observed outcomes. J Neurophysiol 2011, 106:1558-1569.
37. Wilson R.C., Niv Y. Inferring relevance in a changing world. Front Hum Neurosci 2011, 5:189.
38. Kumaran D., Maguire E.A. Novelty signals: a window into hippocampal information processing. Trends Cogn Sci 2009, 13:47-54.
39. Schapiro A.C., Rogers T.T., Cordova N.I., Turk-Browne N.B., Botvinick M.M. Neural representations of events arise from temporal community structure. Nat Neurosci 2013, 16:486-492.
40. Konidaris G., Scheidwasser I., Barto A.G. Transfer in reinforcement learning via shared features. J Mach Learn Res 2012, 13:1333-1371.
41. Mnih V., Kavukcuoglu K., Silver D., Graves A., Antonoglou I., Wierstra D., Riedmiller M. Playing Atari with deep reinforcement learning 2013, arxiv:1312.5602.
42. Barto A.G., Mahadevan S. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 2003, 13:341-379.
43. Sutton R.S., Precup D., Singh S. Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif Intell 1999, 112:181-211.
44. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
45. Badre D., D'Esposito M. Is the rostro-caudal axis of the frontal lobe hierarchical?. Nat Rev Neurosci 2009, 10:659-669.
46. Koechlin E., Ody C., Kouneiher F. The architecture of cognitive control in the human prefrontal cortex. Science 2003, 302:1181-1185.
47. Ribas-Fernandes J.J., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., Botvinick M.M. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71:370-379.
48. Behrens T.E., Woolrich M.W., Walton M.E., Rushworth M.F. Learning the value of information in an uncertain world. Nat Neurosci 2007, 10:1214-1221.
49. O'Reilly J.X., Jbabdi S., Behrens T.E. How can a Bayesian approach inform neuroscience?. Eur J Neurosci 2012, 35:1169-1179.
50. Friston K., Schwartenbeck P., Fitzgerald T., Moutoussis M., Behrens T., Dolan R.J. The anatomy of choice: active inference and agency. Front Hum Neurosci 2013, 7:598.
51. Yu A.J., Dayan P. Expected and unexpected uncertainty. Adv Neural Inform Process Syst 2003, 15.
52. Payzan-LeNestour E., Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput Biol 2011, 7:e1001048.
53. Badre D., Doll B.B., Long N.M., Frank M.J. Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron 2012, 73:595-607.
54. Payzan-Lenestour E., Bossaerts P. Do not bet on the unknown versus try to find out more: estimation uncertainty and 'unexpected uncertainty' both modulate exploration. Front Neurosci 2012, 6:150.
55. Schwartenbeck P., Fitzgerald T., Dolan R.J., Friston K. Exploration, novelty, surprise, and free energy minimization. Front Psychol 2013, 4:710.
56. Donoso M., Collins A.G., Koechlin E. Human cognition, Foundations of human reasoning in the prefrontal cortex. Science 2014, 344:1481-1486.
57. Prevost C., McCabe J.A., Jessup R.K., Bossaerts P., O'Doherty J.P. Differentiable contributions of human amygdalar subregions in the computations underlying reward and avoidance learning. Eur J Neurosci 2011, 34:134-145.
58. Payzan-LeNestour E., Dunne S., Bossaerts P., O'Doherty J.P. The neural representation of unexpected uncertainty during value-based decision making. Neuron 2013, 79:191-201.
59. Prevost C., McNamee D., Jessup R.K., Bossaerts P., O'Doherty J.P. Evidence for model-based computations in the human amygdala during Pavlovian conditioning. PLoS Comput Biol 2013, 9:e1002918.
60. Schwartenbeck P., FitzGerald T.H., Mathys C., Dolan R., Friston K. The dopaminergic midbrain encodes the expected certainty about desired outcomes. Cereb Cortex 2014.
61. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 1980, 87:532-552.
62. Preuschoff K., Bossaerts P. Adding prediction risk to the theory of reward learning. Ann N Y Acad Sci 2007.
63. Li J., Schiller D., Schoenbaum G., Phelps E.A., Daw N.D. Differential roles of human striatum and amygdala in associative learning. Nat Neurosci 2011, 14:1250-1252.
64. Dearden R., Friedman N., Andre D. Model-based Bayesian exploration. Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence 1999, 150-159.
65. Dearden R., Friedman N., Russell S. Bayesian Q-learning. Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence 1998, 761-768.
66. Norman D.A., Shallice T. Attention to action: Willed and automatic control of behavior. Consciousness and Self-Regulation: Advances in Research and Theory (IV), edited by Davidson R et al. 1986, 1-18.
67. Hommel B. Intentional control of automatic stimulus-response translation. Beyond dissociation: Interaction between dissociated implicit and explicit processing 2000, 221-242.
68. Dezfouli A., Balleine B.W. Habits, action sequences and reinforcement learning. Eur J Neurosci 2012, 35:1036-1051.
69. Lee S.W., Kim Y.S., Bien Z. A nonsupervised learning framework of human behavior patterns based on sequential actions. IEEE Trans Knowledge Data Eng 2010, 22:479-492.
70. Lee S.W., Prenzel O., Bien Z. Applying human learning principles to user-centered IoT systems. IEEE Computer 2013, 46:46-52.