Model-based choices involve prospective neural activity

Nature Neuroscience

Volumn 18, Issue 5, 2015, Pages 767-772

  Doll, Bradley B ^a,b   Duncan, Katherine D ^b   Simon, Dylan A ^a   Shohamy, Daphna ^b   Daw, Nathaniel D ^a,b  

^a NEW YORK UNIVERSITY   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ANALYTICAL ERROR; ARTICLE; BEHAVIOR; BOLD SIGNAL; DECISION MAKING; ELECTROENCEPHALOGRAM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; LEARNING; MODEL; NORMAL HUMAN; PREDICTION; PRIORITY JOURNAL; REWARD; ANTICIPATION; BIOLOGICAL MODEL; FORECASTING; FUNCTIONAL NEUROIMAGING; GAME; MALE; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; YOUNG ADULT;

ADULT; ANTICIPATION, PSYCHOLOGICAL; CHOICE BEHAVIOR; DECISION MAKING; FEMALE; FORECASTING; FUNCTIONAL NEUROIMAGING; GAMES, EXPERIMENTAL; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; MODELS, NEUROLOGICAL; REWARD; YOUNG ADULT;

EID: 84928701322     PISSN: 10976256     EISSN: 15461726     Source Type: Journal    
DOI: 10.1038/nn.3981     Document Type: Article

References (50)

1. Thorndike, E.L. Animal Intelligence: Experimental Studies (Macmillan, New York, 1911).
2. Sutton, R.S. & Barto, A.G. Introduction to Reinforcement Learning 〈http://dl.acm.org/citation.cfm?id=551283〉 (MIT Press, 1998).
3. Tolman, E.C. Cognitive maps in rats and men. Psychol. Rev. 55, 189-208 (1948).
4. Shohamy, D. & Wagner, A.D. Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 60, 378-389 (2008).
5. Wimmer, G.E. & Shohamy, D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 338, 270-273 (2012).
6. Barron, H.C., Dolan, R.J. & Behrens, T.E.J. Online evaluation of novel choices by simultaneous representation of multiple memories. Nat. Neurosci. 16, 1492-1498 (2013).
7. Doll, B.B., Simon, D.A. & Daw, N.D. The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 22, 1075-1081 (2012).
8. Dolan, R.J. & Dayan, P. Goals and habits in the brain. Neuron 80, 312-325 (2013).
9. Doya, K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961-974 (1999).
10. Fermin, A., Yoshida, T., Ito, M., Yoshimoto, J. & Doya, K. Evidence for model-based action planning in a sequential fnger movement task. J. Mot. Behav. 42, 371-379 (2010).
11. Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585-595 (2010).
12. Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. Model-based infuences on humans' choices and striatal prediction errors. Neuron 69, 1204-1215 (2011).
13. Eppinger, B., Walter, M., Heekeren, H.R. & Li, S.-C. Of goals and habits: age-related and individual differences in goal-directed decision-making. Front. Neurosci. 7, 253 (2013).
14. Pfeiffer, B.E. & Foster, D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74-79 (2013).
15. Johnson, A. & Redish, A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 27, 12176-12189 (2007).
16. Schapiro, A.C., Kustner, L.V. & Turk-Browne, N.B. Shaping of object representations in the human medial temporal lobe based on temporal regularities. Curr. Biol. 22, 1622-1627 (2012).
17. Moore, A.W. & Atkeson, C.G. Prioritized sweeping: reinforcement learning with less data and less time. Mach. Learn. 13, 103-130 (1993).
18. Sutton, R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Machine Learning: Proc. Seventh Int. Conf. on Machine Learning (eds. Porter, B.W. & Mooney, R.J.) 216-224 (Morgan Kaufmann, Palo Alto, California, USA, 1990).
19. Daw, N.D. & Dayan, P. The algorithmic anatomy of model-based evaluation. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369, 20130478 (2014).
20. Zeithamova, D., Dominick, A.L. & Preston, A.R. Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 75, 168-179 (2012).
21. Gershman, S.J., Markman, A.B. & Otto, A.R. Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. 143, 182-194 (2014).
22. Doll, B.B., Shohamy, D. & Daw, N.D. Multiple memory systems as substrates for multiple decision systems. Neurobiol. Learn. Mem. 117, 4-13 (2015).
23. Lee, S.W., Shimojo, S. & O'Doherty, J.P. Neural computations underlying arbitration between model-based and model-free learning. Neuron 81, 687-699 (2014).
24. Reddy, L. & Kanwisher, N. Coding of visual objects in the ventral stream. Curr. Opin. Neurobiol. 16, 408-414 (2006).
25. FitzGerald, T.H.B., Seymour, B. & Dolan, R.J. The role of human orbitofrontal cortex in value comparison for incommensurable objects. J. Neurosci. 29, 8388-8395 (2009).
26. Boorman, E.D., Behrens, T.E.J., Woolrich, M.W. & Rushworth, M.F.S. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62, 733-743 (2009).
27. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876-879 (2006).
28. Boorman, E.D., Behrens, T.E. & Rushworth, M.F. Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol. 9, e1001093 (2011).
29. Kolling, N., Behrens, T.E.J., Mars, R.B. & Rushworth, M.F.S. Neural mechanisms of foraging. Science 336, 95-98 (2012).
30. Shenhav, A., Straccia, M.A., Cohen, J.D. & Botvinick, M.M. Anterior cingulate engagement in a foraging context refects choice diffculty, not foraging value. Nat. Neurosci. 17, 1249-1254 (2014).
31. Garrison, J., Erdeniz, B. & Done, J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci. Biobehav. Rev. 37, 1297-1310 (2013).
32. Foerde, K., Knowlton, B.J. & Poldrack, R.A. Modulation of competing memory systems by distraction. Proc. Natl. Acad. Sci. USA 103, 11778-11783 (2006).
33. Tricomi, E., Balleine, B.W. & O'Doherty, J.P. A specifc role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225-2232 (2009).
34. Wunderlich, K., Dayan, P. & Dolan, R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 15, 786-791 (2012).
35. Kurth-Nelson, Z., Barnes, G., Sejdinovic, D., Dolan, R. & Dayan, P. Temporal structure in associative retrieval. Elife 4, e04919 (2015).
36. Tolman, E.C. & Honzik, C.H. Introduction and removal of reward, and maze performance in rats. Univ. Calif. Publ. Psychol. 4, 257-275 (1930).
37. Daw, N.D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704-1711 (2005).
38. Dayan, P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613-624 (1993).
39. Botvinick, M. & Weinstein, A. Model-based hierarchical reinforcement learning and human action control. Philos. Trans. R. Soc. Lond. B Biol. Sci. 369, 20130480 (2014).
40. Schapiro, A.C., Rogers, T.T., Cordova, N.I., Turk-Browne, N.B. & Botvinick, M.M. Neural representations of events arise from temporal community structure. Nat. Neurosci. 16, 486-492 (2013).
41. Gluck, M.A. & Myers, C.E. Hippocampal mediation of stimulus representation: a computational theory. Hippocampus 3, 491-516 (1993).
42. Badre, D., Kayser, A.S. & D'Esposito, M. Frontal cortex and the discovery of abstract action rules. Neuron 66, 315-326 (2010).
43. Botvinick, M.M., Niv, Y. & Barto, A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113, 262-280 (2009).
44. Simon, D.A. & Daw, N.D. Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. 31, 5526-5539 (2011).
45. Everitt, B.J. & Robbins, T.W. Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 8, 1481-1489 (2005).
46. Redish, A.D. Addiction as a computational process gone awry. Science 306, 1944-1947 (2004).
47. Voon, V. et al. Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron 65, 135-142 (2010).
48. Otto, A.R., Gershman, S.J., Markman, A.B. & Daw, N.D. The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol. Sci. 24, 751-761 (2013).
49. Akaike, H. A new look at the statistical model identifcation. IEEE Trans. Automat. Contr. 19, 716-723 (1974).
50. Daw, N.D. in Atten. Perform. XXIII (Delgado, M.R., Phelps, E.A. & Robbins, T.W.) 1-26 (Oxford University Press, 2011).