Neural Computations Underlying Arbitration between Model-Based and Model-free Learning

Neuron

Volumn 81, Issue 3, 2014, Pages 687-699

  Wan Lee, Sang ^a   Shimojo, Shinsuke ^a   O'Doherty, John P ^a  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACCOMMODATION; ADULT; ANALYTICAL PARAMETERS; ARTICLE; ARTIFICIAL NEURAL NETWORK; BRAIN CORTEX; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; CORRELATION ANALYSIS; FEMALE; FRONTOPOLAR CORTEX; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUNCTIONAL NEUROIMAGING; HUMAN; HUMAN EXPERIMENT; LEARNING; MACHINE LEARNING; MALE; MATHEMATICAL MODEL; NEUROMODULATION; NORMAL HUMAN; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; PROBABILITY; REACTION TIME; REINFORCEMENT; RELIABILITY; REWARD PREDICTION ERROR; SIGNAL PROCESSING; SIGNAL TRANSDUCTION; STATE PREDICTION ERROR; YOUNG ADULT;

ADULT; BRAIN; BRAIN MAPPING; COMPUTER SIMULATION; DECISION MAKING; FEMALE; GOALS; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; OXYGEN; PSYCHOPHYSICS; YOUNG ADULT;

EID: 84893508813     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2013.11.028     Document Type: Article

References (58)

1. Adams C.D., Dickinson A. Instrumental responding following reinforcer devaluation. Quarterly journal of experimental psychology. Q.J. Exp. Psychol. 1981, 33:109-122.
2. Aron A.R., Fletcher P.C., Bullmore E.T., Sahakian B.J., Robbins T.W. Stop-signal inhibition disrupted by damage to right inferior frontal gyrus in humans. Nat. Neurosci. 2003, 6:115-116.
3. Aron A.R., Robbins T.W., Poldrack R.A. Inhibition and the right inferior frontal cortex. Trends Cogn. Sci. 2004, 8:170-177.
4. Badre D., Doll B.B., Long N.M., Frank M.J. Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron 2012, 73:595-607.
5. Baird B., Smallwood J., Gorgolewski K.J., Margulies D.S. Medial and lateral networks in anterior prefrontal cortex support metacognitive ability for memory and perception. J.Neurosci 2013, 33:16657-16665.
6. Balleine B.W., Dickinson A. Goal-directed instrumental action:contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
7. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35:48-69.
8. Beck J.M., Ma W.J., Kiani R., Hanks T., Churchland A.K., Roitman J., Shadlen M.N., Latham P.E., Pouget A. Probabilistic population codes for Bayesian decision making. Neuron 2008, 60:1142-1152.
9. Beierholm U.R., Anen C., Quartz S., Bossaerts P. Separate encoding of model-based and model-free valuations in the human brain. Neuroimage 2011, 58:955-962.
10. Boorman E.D., Behrens T.E., Woolrich M.W., Rushworth M.F.S. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 2009, 62:733-743.
11. Burguière E., Monteiro P., Feng G., Graybiel A.M. Optogenetic stimulation of lateral orbitofronto-striatal pathway suppresses compulsive behaviors. Science 2013, 340:1243-1246.
12. Cools R., Clark L., Owen A., Robbins T.W. Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. J.Neurosci. 2002, 22:4563-4567.
13. Coutureau E., Killcross S. Inactivation of the infralimbic prefrontal cortex reinstates goal-directed responding in overtrained rats. Behav. Brain Res. 2003, 146:167-174.
14. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
15. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
16. De Martino B., Fleming S.M., Garrett N., Dolan R.J. Confidence in value-based choice. Nat. Neurosci 2013, 16:105-110.
17. de Wit S., Corlett P.R., Aitken M.R., Dickinson A., Fletcher P.C. Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans. J.Neurosci. 2009, 29:11330-11338.
18. de Wit S., Watson P., Harsay H.A., Cohen M.X., van de Vijver I., Ridderinkhof K.R. Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J.Neurosci. 2012, 32:12066-12075.
19. Doya K., Samejima K., Katagiri K., Kawato M. Multiple model-based reinforcement learning. Neural Comput. 2002, 14:1347-1369.
20. Esterman M., Tamber-Rosenau B.J., Chiu Y.C., Yantis S. Avoiding non-independence in fMRI data analysis: leave one subject out. Neuroimage 2010, 50:572-576.
21. Everitt B.J., Robbins T.W. Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 2005, 8:1481-1489.
22. Friston K.J., Buechel C., Fink G.R., Morris J., Rolls E.T., Dolan R.J. Psychophysiological and modulatory interactions in neuroimaging. Neuroimage 1997, 6:218-229.
23. Garavan H., Ross T.J., Stein E.A. Right hemispheric dominance of inhibitory control: an event-related functional MRI study. Proc. Natl. Acad. Sci. USA 1999, 96:8301-8306.
24. Gillan C.M., Papmeyer M., Morein-Zamir S., Sahakian B.J., Fineberg N.A., Robbins T.W., de Wit S. Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. Am. J. Psychiatry 2011, 168:718-726.
25. Gläscher J., Daw N.D., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
26. Graybiel A.M. Habits, rituals, and the evaluative brain. Annu. Rev. Neurosci. 2008, 31:359-387.
27. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J.Neurosci. 2006, 26:8360-8367.
28. Hare T.A., Camerer C.F., Rangel A. Self-control in decision-making involves modulation of the vmPFC valuation system. Science 2009, 324:646-648.
29. Hare T.A., Schultz W., Camerer C.F., O'Doherty J.P., Rangel A. Transformation of stimulus value signals into motor commands during simple choice. Proc. Natl. Acad. Sci. USA 2011, 108:18120-18125.
30. Koechlin E., Hyafil A. Anterior prefrontal function and the limits of human decision-making. Science 2007, 18(80):594-598.
31. Li J., Schiller D., Schoenbaum G., Phelps E.A., Daw N.D. Differential roles of human striatum and amygdala in associative learning. Nat. Neurosci. 2011, 14:1250-1252.
32. Luce R.D. Individual choice behavior: a theoretical analysis 1959, Wiley, New York.
33. Ma W.J., Beck J.M., Latham P.E., Pouget A. Bayesian inference with probabilistic population codes. Nat. Neurosci. 2006, 9:1432-1438.
34. McClure S.M., Berns G.S., Montague P.R. Temporal prediction errors in a passive learning task activate human striatum. Neuron 2003, 38:339-346.
35. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J.Neurosci. 1996, 16:1936-1947.
36. O'Doherty J.P. Contributions of the ventromedial prefrontal cortex to goal-directed action selection. Ann. N Y Acad. Sci. 2011, 1239:118-129.
37. O'Doherty J.P., Dayan P., Friston K., Critchley H., Dolan R.J. Temporal difference models and reward-related learning in the human brain. Neuron 2003, 38:329-337.
38. O'Doherty J.P., Dayan P., Schultz J., Deichmann R., Friston K., Dolan R.J. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 2004, 304:452-454.
39. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532-552.
40. Pennini F., Plastino A. Diverging Fano factors. J.Phys. Conf. Ser. 2010, 246. 10.1088/1742-6596/246/1/012030.
41. Preuschoff K., Quartz S.R., Bossaerts P. Human insula activation reflects risk prediction errors as well as risk. J.Neurosci. 2008, 28:2745-2752.
42. Roesch M.R., Calu D.J., Esber G.R., Schoenbaum G. Neural correlates of variations in event processing during learning in basolateral amygdala. J.Neurosci. 2010, 30:2464-2471.
43. Rowe J.B., Hughes L., Nimmo-Smith I. Action selection: a race model for selected and non-selected actions distinguishes the contribution of premotor and prefrontal areas. Neuroimage 2010, 51:888-896.
44. Rushworth M.F.S., Noonan M.P., Boorman E.D., Walton M.E., Behrens T.E. Frontal cortex and reward-guided learning and decision-making. Neuron 2011, 70:1054-1069.
45. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
46. Smith K.S., Virkud A., Deisseroth K., Graybiel A.M. Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proc. Natl. Acad. Sci. USA 2012, 109:18932-18937.
47. Staudinger M.R., Büchel C. How initial confirmatory experience potentiates the detrimental influence of bad advice. Neuroimage 2013, 76:125-133.
48. Stephan K.E., Penny W.D., Daunizeau J., Moran R.J., Friston K.J. Bayesian model selection for group studies. Neuroimage 2009, 46:1004-1017.
49. Sutton R.S., Barto A.G. Reinforcement Learning 1998, MIT press, Cambridge.
50. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 2011, 14:1590-1597.
51. Tanaka S.C., Balleine B.W., O'Doherty J.P. Calculating consequences: brain systems that encode the causal effects of actions. J.Neurosci. 2008, 28:6750-6755.
52. Tanji J., Hoshi E. Role of the lateral prefrontal cortex in executive behavioral control. Physiol. Rev. 2008, 88:37-57.
53. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 2009, 29:2225-2232.
54. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J.Neurosci. 2007, 27:4019-4026.
55. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15:786-791.
56. Xue G., Ghahremani D.G., Poldrack R.A. Neural substrates forreversing stimulus-outcome and stimulus-response associations. J.Neurosci. 2008, 28:11196-11204.
57. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 2004, 19:181-189.
58. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 2005, 22:513-523.