The ubiquity of model-based reinforcement learning

Current Opinion in Neurobiology

Volumn 22, Issue 6, 2012, Pages 1075-1081

  Doll, Bradley B ^a,b   Simon, Dylan A ^b   Daw, Nathaniel D ^b  

^a Och Spine at New York Presbyterian Hospitals   (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; BIOLOGICAL MODEL; BRAIN; DECISION MAKING; HUMAN; PHYSIOLOGY; REINFORCEMENT; REVIEW; REWARD;

ANIMALS; BRAIN; CHOICE BEHAVIOR; DECISION MAKING; HUMANS; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 84872761547     PISSN: 09594388     EISSN: 18736882     Source Type: Journal    
DOI: 10.1016/j.conb.2012.08.003     Document Type: Review

References (60)

1. Barto A.G. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1995, 215-232. MIT Press, Cambridge, MA, (Chapter xii). J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
2. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 1996, 6:1936-1947.
3. Thorndike E.L. Animal intelligence: an experimental study of the associative processes in animals. Psychol Rev Monogr Suppl 1898, 2:1-8.
4. Tolman E.C. Cognitive maps in rats and men. Psychol Rev 1948, 55:189-208.
5. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711. 10.1038/nn1560.
6. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?. Neural Netw 1999, 12:961-974. 10.1016/S0893-6080(99)00046-5.
7. Solway A., Botvinick M.M. Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol Rev 2012, 119:120-154. 10.1037/a0026435.
8. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2009, 35:48-69. 10.1038/npp.2009.131.
9. Keramati M., Dezfouli A., Piray P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 2011, 7:e1002055. 10.1371/journal.pcbi.1002055.
10. Adams C.D. Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q J Exp Psychol B 1982, 34:77-98. 10.1080/14640748208400878.
11. Derusso A.L., Fan D., Gupta J., Shelest O., Costa R.M., Yin H.H. Instrumental uncertainty as a determinant of behavior under interval schedules of reinforcement. Front Integr Neurosci 2010, 4. 10.3389/fnint.2010.00017.
12. Simon D.A., Daw N.D. Environmental statistics and the trade-off between model-based and TD learning in humans. Advances in Neural Information Processing Systems, vol 24 2011, 127-135. J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, K. Weinberger (Eds.).
13. Beierholm U.R., Anen C., Quartz S., Bossaerts P. Separate encoding of model-based and model-free valuations in the human brain. Neuroimage 2011, 58:955-962. 10.1016/j.neuroimage.2011.06.071.
14. Dias-Ferreira E., Sousa J.a.C., Melo I., Morgado P., Mesquita A.R., Cerqueira J.a.J., Costa R.M., Sousa N. Chronic stress causes frontostriatal reorganization and affects decision-making. Science 2009, 325:621-625. 10.1126/science.1171203.
15. Schwabe L., Wolf O.T. Socially evaluated cold pressor stress after instrumental learning favors habits over goal-directed action. Psychoneuroendocrinology 2010, 35:977-986. 10.1016/j.psyneuen.2009.12.010.
16. Schwabe L., Wolf O.T. Stress-induced modulation of instrumental behavior: from goal-directed to habitual control of action. Behav Brain Res 2011, 219:321-328. 10.1016/j.bbr.2010.12.038.
17. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur J Neurosci 2009, 29:2225-2232. 10.1111/j.1460-9568.2009.06796.x.
18. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 2004, 19:181-189. 10.1111/j.1460-9568.2004.03095.x.
19. Tsai H-C., Zhang F., Adamantidis A., Stuber G.D., Bonci A., de Lecea L., Deisseroth K. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 2009, 324:1080-1084. 10.1126/science.1168878.
20. Zweifel L.S., Parker J.G., Lobb C.J., Rainwater A., Wall V.Z., Fadok J.P., Darvas M., Kim M.J., Mizumori S.J.Y., Paladini C.A., et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc Natl Acad Sci U S A 2009, 106:7281-7288. 10.1073/pnas.0813415106.
21. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. Eur J Neurosci 2005, 22:513-523. 10.1111/j.1460-9568.2005.04218.x.
22. Glascher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595. 10.1016/j.neuron.2010.04.016.
23. Haber S.N., Knutson B. The reward circuit: linking primate anatomy and human imaging. Neuropsychopharmacology 2009, 35:4-26. 10.1038/npp.2009.129.
24. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841. 10.1038/nature08028.
25. Brischoux F., Chakraborty S., Brierley D.I., Ungless M.A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A 2009, 106:4894-4899. 10.1073/pnas.0811507106.
26. Ungless M.A., Grace A.A. Are you or aren't you? Challenges associated with physiologically identifying dopamine neurons. Trends Neurosci 2012, 35:422-430. 10.1016/j.tins.2012.02.003.
27. Zapata A., Minney V.L., Shippenberg T.S. Shift from goal-directed to habitual cocaine seeking after prolonged experience in rats. J Neurosci 2010, 30:15457-15463. 10.1523/JNEUROSCI.4072-10.2010.
28. Faure A., Haberland U., Coné F., Massioui N.E. Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J Neurosci 2005, 25:2771-2780. 10.1523/JNEUROSCI.3894-04.2005.
29. Wang L.P., Li F., Wang D., Xie K., Wang D., Shen X., Tsien J.Z. NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron 2011, 72:1055-1066. 10.1016/j.neuron.2011.10.019.
30. Dickinson A., Smith J., Mirenowicz J. Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav Neurosci 2000, 114:468-483. 10.1037/0735-7044.114.3.468.
31. de Wit S., Barker R.A., Dickinson A.D., Cools R. Habitual versus goal-directed action control in Parkinson disease. J Cogn Neurosci 2011, 23:1218-1229. 10.1162/jocn.2010.21514.
32. de Wit S., Standing H.R., Devito E.E., Robinson O.J., Ridderinkhof K.R., Robbins T.W., Sahakian B.J. Reliance on habits at the expense of goal-directed control following dopamine precursor depletion. Psychopharmacology (Berl) 2012, 219:621-631. 10.1007/s00213-011-2563-2.
33. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215. 10.1016/j.neuron.2011.02.027.
34. Simon D.A., Daw N.D. Neural correlates of forward planning in a spatial decision task in humans. J Neurosci 2011, 31:5526-5539. 10.1523/JNEUROSCI.4647-10.2011.
35. van der Meer M.A.A., Johnson A., Schmitzer-Torbert N.C., Redish A.D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32. 10.1016/j.neuron.2010.06.023.
36. van der Meer M.A.A., Redish A.D. Expectancies in decision making, reinforcement learning, and ventral striatum. Front Neurosci 2010, 4:6. 10.3389/neuro.01.006.2010.
37. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 2012, 15:786-791. 10.1038/nn.3068.
38. Fermin A., Yoshida T., Ito M., Yoshimoto J., Doya K. Evidence for model-based action planning in a sequential finger movement task. J Mot Behav 2010, 42:371-379. 10.1080/00222895.2010.526467.
39. Pan X., Sawa K., Tsuda I., Tsukada M., Sakagami M. Reward prediction based on stimulus categorization in primate lateral prefrontal cortex. Nat Neurosci 2008, 11:703-712. 10.1038/nn.2128.
40. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J Neurosci 2006, 26:8360-8367. 10.1523/JNEUROSCI.1010-06.2006.
41. Bromberg-Martin E.S., Matsumoto M., Hong S., Hikosaka O. A pallidus-habenula-dopamine pathway signals inferred stimulus values. J Neurophysiol 2010, 104:1068-1076. 10.1152/jn.00158.2010.
42. Abe H., Lee D. Distributed coding of actual and hypothetical outcomes in the orbital and dorsolateral prefrontal cortex. Neuron 2011, 70:731-741. 10.1016/j.neuron.2011.03.026.
43. Wunderlich K., Symmonds M., Bossaerts P., Dolan R.J. Hedging your bets by learning reward correlations in the human brain. Neuron 2011, 71:1141-1152. 10.1016/j.neuron.2011.07.025.
44. Wimmer G.E., Daw N.D., Shohamy D. Generalization of value in reinforcement learning by humans. Eur J Neurosci 2012, 35:1092-1104. 10.1111/j.1460-9568.2012.08017.x.
45. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J Neurosci 2007, 27:4019-4026. 10.1523/JNEUROSCI.0564-07.2007.
46. de Wit S., Corlett P.R., Aitken M.R., Dickinson A., Fletcher P.C. Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. J Neurosci 2009, 29:11330-11338. 10.1523/JNEUROSCI.1639-09.2009.
47. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 2011, 31:2700-2705. 10.1523/JNEUROSCI.5499-10.2011.
48. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441:223-226. 10.1038/nature04676.
49. Wallis J.D. Cross-species studies of orbitofrontal cortex and value-based decision-making. Nat Neurosci 2012, 15:13-19. 10.1038/nn.2956.
50. O'Keefe J., Nadel L. The Hippocampus as a Cognitive Map 1978, Clarendon Press, Oxford.
51. Bornstein A.M., Daw N.D. Dissociating hippocampal and striatal contributions to sequential prediction learning. Eur J Neurosci 2012, 35:1011-1023. 10.1111/j.1460-9568.2011.07920.x.
52. Frank M.J., Badre D. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits. 1: computational analysis. Cereb Cortex 2012, 22:509-526. 10.1093/cercor/bhr114.
53. Dayan P. Improving generalization for temporal difference learning: the successor representation. Neural Comput 1993, 5:613-624. 10.1162/neco.1993.5.4.613.
54. Doll B.B., Hutchison K.E., Frank M.J. Dopaminergic genes predict individual differences in susceptibility to confirmation bias. J Neurosci 2011, 31:6188-6198. 10.1523/JNEUROSCI.6486-10.2011.
55. Biele G., Rieskamp J., Krugel L.K., Heekeren H.R. The neural basis of following advice. PLoS Biol 2011, 9:e1001089. 10.1371/journal.pbio.1001089.
56. Doll B.B., Jacobs W.J., Sanfey A.G., Frank M.J. Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res 2009, 1299:74-94. 10.1016/j.brainres.2009.07.007.
57. Simon D.A., Daw N.D. Dual-system learning models and drugs of abuse. Computational Neuroscience of Drug Addiction, vol 10 of Computational Neuroscience 2012, 145-162. Springer, (Chapter 5). B. Gutkin, S.H. Ahmed (Eds.).
58. Sutton R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning 1990, 216-224. GTE Laboratories Incorporated, Morgan Kaufmann.
59. Walsh T.J., Goschin S., Littman M.L. Integrating sample-based planning and model-based reinforcement learning. AAAI Conference on Artificial Intelligence, vol 24 2010.
60. Huys Q.J.M., Eshel N., O'Nions E., Sheridan L., Dayan P., Roiser J.P. Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput Biol 2012, 8:e1002410. 10.1371/journal.pcbi.1002410.