Cortical and Hippocampal Correlates of Deliberation during Model-Based Decisions for Rewards in Humans

PLoS Computational Biology

Volumn 9, Issue 12, 2013, Pages

  Bornstein, Aaron M ^a,b   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

^b PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BRAIN; NEURONS;

COGNITIVE MAPS; CORTICAL STRUCTURES; DUAL REPRESENTATION; FULL CYCLES; LEARN+; LEARNING PROCESS; MEMORY SYSTEMS; MODEL-BASED DECISION-MAKING; MODEL-BASED OPC; RELATIONAL LEARNING;

DECISION MAKING;

ADULT; ARTICLE; BRAIN CORTEX; BRAIN REGION; CONTROLLED STUDY; DECISION MAKING; FEMALE; FUNCTIONAL ANATOMY; FUNCTIONAL MAGNETIC RESONANCE IMAGING; GOAL ATTAINMENT; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; LEARNING ENVIRONMENT; MALE; MATHEMATICAL MODEL; MEMORY CONSOLIDATION; MENTAL TASK; NEUROANATOMY; NORMAL HUMAN; RECALL; RESPONSE TIME; REWARD; STIMULUS RESPONSE; YOUNG ADULT;

CEREBRAL CORTEX; DECISION MAKING; HIPPOCAMPUS; HUMANS; MAGNETIC RESONANCE IMAGING; MODELS, PSYCHOLOGICAL; MOTIVATION;

EID: 84892777219     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1003387     Document Type: Article

References (101)

1. Dickinson A, Balleine BW (2002) The role of learning in the operation of motivational systems. In: Gallistel CR, Pashler HV, editors. Stevens Handbook of Experimental Psychology. Vol. 3: Learning, Motivation and Emotion. New York, NY: John Wiley & Sons Inc. pp. 497-533.
2. Dickinson A (1980) Contemporary Animal Learning Theory. Cambridge: Cambridge University Press.
3. Daw ND, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8: 1704-1711.
4. Thorndike EL (1911) Animal Intelligence. New York: Macmillan.
5. Barto AC (1995) Adaptive Critics and the Basal Ganglia. In: Houk JC, Davis JL, Beiser DG, editors. Models of information processing in the basal ganglia, Cambridge, MA: MIT Press. pp. 215-232.
6. Schultz W, Montague PR, Dayan P, (1997) A Neural Substrate of Prediction and Reward. Science 275: 1593-1599.
7. Bornstein AM, Daw ND, (2012) Dissociating hippocampal and striatal contributions to sequential prediction learning. European Journal of Neuroscience 35: 1011-1023.
8. Bahrick H, (1954) Incidental learning under two incentive conditions. Journal of Experimental Psychology 47: 170-172.
9. Strange BA, Duggins A, Penny W, Dolan RJ, Friston KJ, (2005) Information theory, novelty and hippocampal responses: unpredicted or unpredictable? Neural Networks 18: 225-230.
10. Harrison LM, Duggins A, Friston KJ, (2006) Encoding uncertainty in the hippocampus. Neural Networks 19: 535-546.
11. Bestmann S, Harrison L, Blankenburg F, Mars R, Haggard P, et al. (2008) Influence of contextual uncertainty and surprise on human corticospinal excitability during preparation for action. Current Biology 18: 775-80.
12. Turk-Browne N, Scholl B, Johnson M, Chun M, (2009) Neural evidence of statistical learning: efficient detection of visual regularities without awareness. Journal of Cognitive Neuroscience 21: 1934-45.
13. Turk-Browne N, Scholl B, Johnson M, Chun M, (2010) Implicit Perceptual Anticipation Triggered by Statistical Learning. Journal of Neuroscience 30: 11177-87.
14. Tolman EC, (1948) Cognitive Maps in Rats and Men. Psychological Review 55: 189-208.
15. Gläscher J, Daw N, Dayan P, O'Doherty JP, (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66: 585-95.
16. Bush RR, Mosteller F, (1956) A Stochastic Model with Applications to Learning. The Annals of Mathematical Statistics 24: 559-585.
17. Rescorla RA, Wagner AR (1972) A Theory of Pavlovian Conditioning: Variations in the Effectiveness of Reinforcement and Nonreinforcement. In: Black AH, Prokasy WF, editors. Classical Conditioning II: Current research and theory. New York: Appleton-Century-Crofts. pp. 64-99.
18. Ratcliff R, (1978) A Theory of Memory Retrieval. Psychological Review 85: 59-108.
19. Lengyel M, Dayan P, (2008) Hippocampal Contributions to Control: The Third Way. Advances in Neural Information Processing Systems 20: 889-896.
20. Erev I, Glozman I, Hertwig R, (2008) What impacts the impact of rare events. Journal of Risk and Uncertainty 36: 153-177.
21. Gold JI, Shadlen MN, (2002) Banburismus and the Brain: Decoding the Relationship between Sensory Stimuli, Decisions, and Reward. Neuron 36: 299-308.
22. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ, (2006) Cortical substrates for exploratory decisions in humans. Nature 441: 876-879.
23. Schapiro AC, Kustner LV, Turk-Browne NB, (2012) Shaping of object representations in the human medial temporal lobe based on temporal regularities. Current Biology 22: 1622-7.
24. Josephs O, Turner R, Friston K, (1997) Event-Related fMRI. Human Brain Mapping 5: 243-248.
25. Daw ND (2010) Trial-by-trial data analysis using computational models. In: Phelps E, Robbins T, Delgado M, editors. Affect, Learning and Decision Making, Attention and Performance. Xxiii edition. Oxford University Press.
26. Buckner RL, Carroll DC, (2006) Self-projection and the brain. Trends in Cognitive Sciences 11: 49-57.
27. Buckner RL, Andrews-Hanna JR, Schacter DL, (2008) The brain's default network: anatomy, function, and relevance to disease. Annals of the New York Academy of Sciences 1124: 1-38.
28. Kahn I, Andrews-Hanna JR, Vincent JL, Snyder AZ, Buckner RL, (2008) Distinct Cortical Anatomy Linked to Subregions of the Medial Temporal Lobe Revealed by Intrinsic Functional Connectivity. Journal of Neurophysiology 100: 129-139.
29. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA, (2000) Tracking the Hemodynamic Responses to Reward and Punishment in the Striatum. Journal of Neurophysiology 84: 3072-3077.
30. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ, (2003) Temporal Difference Models and Reward-Related Learning in the Human Brain. Neuron 38: 329-337.
31. McClure SM, Berns GS, Montague PR, (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38: 339-46.
32. Downing PE, Jiang Y, Shuman M, Kanwisher N, (2001) A Cortical Area Selective for Visual Processing of the Human Body. Science 293: 2470-2473.
33. Kanwisher N, Mcdermott J, Chun MM, (1997) The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception. Journal of Neuroscience 17: 4302-4311.
34. Epstein R, Kanwisher N, (1998) A cortical representation of the local visual environment. Nature 392: 598-601.
35. Malach R, Reppas JB, Benson RR, Kwong KK, Jlang H, et al. (1995) Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences 92: 8135-8139.
36. Hampton AN, Bossaerts P, O'Doherty JP, (2006) The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience 26: 8360.
37. Hampton AN, Bossaerts P, O'Doherty JP, (2008) Neural correlates of mentalizing-related computations during strategic interactions in humans. Proceedings of the National Academy of Sciences 105: 6741-6746.
38. Daw ND, Gershman SJ, Seymour B, Dayan P, Raymond J, (2011) Model-based influences on humans choices and striatal prediction errors. Neuron 69: 1204-1215.
39. Büchel C, Wise RJ, Mummery CJ, Poline JB, Friston KJ, (1996) Nonlinear regression in parametric activation studies. NeuroImage 4: 60-6.
40. Wittmann BC, Daw ND, Seymour B, Dolan RJ, (2008) Striatal activity underlies novelty-based choice in humans. Neuron 58: 967-73.
41. Wimmer GE, Daw ND, Shohamy D, (2012) Generalization of value in reinforcement learning by humans. The European Journal of Neuroscience 35: 1092-104.
42. Squire LR, (1992) Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans. Psychological Review 99: 195-231.
43. Cohen N, Eichenbaum H (1993) Amnesia, Memory and the Hippocampal System. Cambridge, MA: MIT Press.
44. Rose M, Haider H, Salari N, Buchel C, (2011) Functional Dissociation of Hippocampal Mechanism during Implicit Learning Based on the Domain of Associations. Journal of Neuroscience 31: 13739-13745.
45. Johnson A, Redish AD, (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. Journal of Neuroscience 27: 12176-89.
46. Addis DR, Wong AT, Schacter DL, (2007) Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia 45: 1363-77.
47. Daw ND, Shohamy D, (2008) The Cognitive Neuroscience of Motivation and Learning. Social Cognition 26: 593-620.
48. Buckner RL, (2010) The role of the hippocampus in prediction and imagination. Annual Review of Psychology 61: 27-48.
49. O'Keefe J, Nadel L (1978) The hippocampus as cognitive map. Cambridge: Cambridge University Press.
50. Redish AD (1999) Beyond the cognitive map: From place cells to episodic memory. Cambridge, MA: MIT Press.
51. Bunsey M, Eichenbaum H, (1996) Conservation of hippocampal memory function in rats and humans. Nature 379: 255-257.
52. Dusek JA, Eichenbaum H, (1997) The hippocampus and memory for orderly stimulus relations. Proceedings of the National Academy of Sciences 94: 7109-7114.
53. Shohamy D, Wagner AD, (2008) Integrating memories in the human brain: Hippocampal-midbrain encoding of overlapping event. Neuron 60: 378-89.
54. Kumaran D, Summerfield JJ, Hassabis D, Maguire EA, (2009) Tracking the emergence of conceptual knowledge during human decision making. Neuron 63: 889-901.
55. Kumaran D, Melo HL, Duzel E, (2012) The emergence and representation of knowledge about social and nonsocial hierarchies. Neuron 76: 653-66.
56. Wimmer G, Shohamy D, (2012) Preference by association: How memory mechanisms in the hippocampus bias decisions. Science 338: 270-3.
57. Simon DA, Daw ND (2011) Environmental statistics and the trade-off between model-based and TD learning in humans. In: Shawe-Taylor J, Zemel RS, Bartlett P, Pereira F, Weinberger K, editors. Advances in Neural Information Processing Systems 24. pp. 127-135.
58. Yin HH, Knowlton BJ, Balleine BW, (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience 19: 181-189.
59. Yin HH, Knowlton BJ, (2006) The role of the basal ganglia in habit formation. Nature Reviews Neuroscience 7: 464-476.
60. Yin HH, Mulcare SP, Hilário MRF, Clouse E, Davis MI, et al. (2009) Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill. Nature Neuroscience 12: 333-341.
61. Davis DGS, Staddon JER, (1990) Memory for reward in probabilistic choice: Markovian and non-Markovian properties. Behaviour 114: 37-64.
62. Mayr U, (1996) Spatial attention and implicit sequence learning: evidence for independent learning of spatial and nonspatial sequences. Journal of Experimental Psychology: Learning, Memory and Cognition 22: 350-364.
63. Willingham DB, (1999) Implicit motor sequence learning is not purely perceptual. Memory & Cognition 27: 561-72.
64. Packard MG, White M, Ha Q, (1989) Differential Effects of Fornix and Caudate Radial Maze Tasks: Evidence for Multiple Nucleus Lesions on Two Memory Systems. Journal of Neuroscience 9: 1465-1472.
65. McDonald RJ, White NM, (1993) A triple dissociation of memory systems: hippocampus, amygdala, and dorsal striatum. Behavioral Neuroscience 107: 3-22.
66. Knowlton BJ, Mangels JA, Squire LR, (1996) A neostriatal habit learning system in humans. Science 273: 1399-402.
67. Poldrack RA, Packard MG, (2003) Competition among multiple memory systems: converging evidence from animal and human brain studies. Neuropsychologia 41: 245-51.
68. Behrens TEJ, Woolrich MW, Walton ME, Rushworth MFS, (2007) Learning the value of information in an uncertain world. Nature Neuroscience 10: 1214-21.
69. Li L, Miller EK, (1993) The Representation of Stimulus Familiarity Temporal Cortex in Anterior Inferior. Journal of Neurophysiology 69: 1918-1929.
70. Wiggs CL, Martin A, (1998) Properties and mechanisms of perceptual priming. Current Opinion in Neurobiology 8: 227-33.
71. McClure SM, Gilzenrat MS, Cohen JD (2005) An exploration-exploitation model based on norepinephrine and dopamine activity. In: Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, pp. 867-874.
72. Summerfield C, Trittschuh EH, Monti JM, Mesulam Mm, Egner T, (2008) Neural repetition suppression reflects fulfilled perceptual expectations. Nature Neuroscience 11: 1004-1006.
73. Philiastides M, Biele G, Heekeren H, (2010) A mechanistic account of value computation in the human brain. Proceedings of the National Academy of Sciences 107: 9430-5.
74. Burgess PW, Dumontheil I, Gilbert SJ, (2007) The gateway hypothesis of rostral prefrontal cortex (area 10) function. Trends in Cognitive Sciences 11: 290-8.
75. Schacter DL, Addis DR, (2007) The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philosophical transactions of the Royal Society of London Series B, Biological sciences 362: 773-86.
76. Viard A, Doeller CF, Hartley T, Bird CM, Burgess N, (2011) Anterior hippocampus and goaldirected spatial decision making. Journal of Neuroscience 31: 4613-21.
77. Guitart-Masip M, Barnes GR, Horner A, Bauer M, Dolan RJ, et al. (2013) Synchronization of medial temporal lobe and prefrontal rhythms in human decision making. The Journal of Neuroscience 33: 442-51.
78. Houk J, Adams J, Barto A (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk JC, Davis JL, Beiser DG, editors. Models of information processing in the Basal Ganglia. Cambridge, MA: MIT Press. pp. 249-270.
79. Frank MJ, Seeberger LC, O'Reilly RC, (2004) By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 306: 1940-1943.
80. Keramati M, Dezfouli A, Piray P, (2011) Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes. PLoS Computational Biology 7: e1002055.
81. Wunderlich K, Dayan P, Dolan RJ, (2012) Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience 15: 786-91.
82. Botvinick M, An J (2008) Goal-directed decision making in prefrontal cortex: A computational framework. In: Koller D, Bengio, Y, Schuurmans D, Bouttou L, Culotta A, editors. Advances in Neural Information Processing Systems. Volume 21. pp. 169-176.
83. Solway A, Botvinick MM, (2012) Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychological Review 119: 120-54.
84. Rangel A, Camerer C, Montague PR, (2008) A framework for studying the neurobiology of valuebased decision making. Nature Reviews Neuroscience 9: 545-56.
85. Krajbich I, Rangel A, (2011) Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions. Proceedings of the National Academy of Sciences 108: 13852-13857.
86. Stewart N, Chater N, Brown GDA, (2006) Decision by sampling. Cognitive Psychology 53: 1-26.
87. Bornstein AM, Daw ND, (2011) Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Current Opinion in Neurobiology 21: 374-80.
88. Peters J, Büchel C, (2010) Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 66: 138-48.
89. Schwarz G, (1978) Estimating the Dimension of a Model. Annals of Statistics 6: 461-464.
90. Brainard DH, (1997) The Psychophysics Toolbox. Spatial Vision 10: 433-6.
91. Holmes AP, Friston KJ, (1998) Generalisability, Random Effects & Population Inference. Neuroimage 7: S754.
92. Simon DA, Daw ND, (2011) Neural Correlates of Forward Planning in a Spatial Decision Task in Humans. Journal of Neuroscience 31: 5526-5539.
93. Schönberg T, Daw ND, Joel D, O'Doherty JP, (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. Journal of Neuroscience 27: 12860-7.
94. Schönberg T, O'Doherty JP, Joel D, Inzelberg R, Segev Y, et al. (2010) Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study. NeuroImage 49: 772-81.
95. Gershman SJ, Pesaran B, Daw ND, (2009) Human reinforcement learning subdivides structured action spaces by learning effector-specific values. Journal of Neuroscience 29: 13524-31.
96. Kass RE, Raftery AE, (1995) Bayes Factors. Journal of the American Statistical Association 90: 773-795.
97. Mackay DJC (2003) Information Theory, Inference, and Learning Algorithms. Cambridge, UK: Cambridge University Press. doi:10.2277/0521642981.
98. Boone EL, Ye K, Smith EP, (2005) Assessment of two approximation methods for computing posterior model probabilities. Computational Statistics & Data Analysis 48: 221-234.
99. Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, et al. (2002) Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. NeuroImage 15: 273-89.
100. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI, (2009) Circular analysis in systems neuroscience: the dangers of double dipping. Nature Neuroscience 12: 535-40.
101. Friston KJ, Josephs O, Rees G, Turner R, (1997) Nonlinear Event-Related Responses in fMRI. Magnetic Resonance Methods 39: 41-52.