Multiple memory systems as substrates for multiple decision systems

Neurobiology of Learning and Memory

Volumn 117, Issue , 2015, Pages 4-13

  Doll, Bradley B ^a,b   Shohamy, Daphna ^b   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

Reinforcement learning; Relational memory

Indexed keywords

ARTICLE; COGNITION; CONTROLLED STUDY; CORPUS STRIATUM; CORRELATION ANALYSIS; DECISION MAKING; DECLARATIVE MEMORY; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; MEMORY; NONHUMAN; NORMAL HUMAN; PROBABILITY; PROCEDURAL MEMORY; REINFORCEMENT; RELATIONAL MEMORY; REWARD; STATE DEPENDENT LEARNING; STATISTICAL ANALYSIS; STOCHASTIC MODEL; TASK PERFORMANCE; ANIMAL; BIOLOGICAL MODEL; BRAIN; LEARNING; MONTE CARLO METHOD; PHYSIOLOGY; PSYCHOLOGICAL MODEL; SPATIAL MEMORY;

ANIMALS; BRAIN; CORPUS STRIATUM; DECISION MAKING; HIPPOCAMPUS; HUMANS; LEARNING; MEMORY; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; MONTE CARLO METHOD; REINFORCEMENT (PSYCHOLOGY); REWARD; SPATIAL MEMORY;

EID: 84921000236     PISSN: 10747427     EISSN: 10959564     Source Type: Journal    
DOI: 10.1016/j.nlm.2014.04.014     Document Type: Article

References (79)

1. Adams C.D. Variations in the sensitivity of instrumental responding to reinforcer devaluation. The Quarterly Journal of Experimental Psychology B 1982, 34:77-98.
2. Addis D.R., Wong A.T., Schacter D.L. Remembering the past and imagining the future: Common and distinct neural substrates during event construction and elaboration. Neuropsychologia 2007, 45:1363-1377.
3. Bornstein A.M., Daw N. Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans. PLoS Computational Biology. 2013, 9(12).
4. Bornstein A.M., Daw N.D. Dissociating hippocampal and striatal contributions to sequential prediction learning. European Journal of Neuroscience 2012, 35:1011-1023.
5. Bunsey M., Eichenbaum H. Conservation of hippocampal memory function in rats and humans. Nature 1996, 379:255-257.
6. Burgess N., Maguire E.A., O'Keefe J. The human hippocampus and spatial and episodic memory. Neuron 2002, 35:625-641.
7. Camerer Ho Experience-weighted attraction learning in coordination games: Probability rules, heterogeneity, and time-variation. Journal of Mathematical Psychology 1998, 42:305-326.
8. Cohen N.J., Eichenbaum H. Memory, amnesia, and the hippocampal system 1993, MIT Press.
9. Cohen N.J., Squire L.R. Preserved learning and retention of pattern-analyzing skill in amnesia: Dissociation of knowing how and knowing that. Science 1980, 210:207-210.
10. Corbit L.H., Balleine B.W. The role of the hippocampus in instrumental conditioning. The Journal of Neuroscience 2000, 20:4233-4239.
11. Corbit L.H., Ostlund S.B., Balleine B.W. Sensitivity to instrumental contingency degradation is mediated by the entorhinal cortex and its efferents via the dorsal hippocampus. The Journal of Neuroscience 2002, 22:10976-10984.
12. Corkin S. Acquisition of motor skill after bilateral medial temporal-lobe excision. Neuropsychologia 1968, 6:225-264.
13. Davis H. Transitive inference in rats (rattus norvegicus). Journal of Comparative Psychology 1992, 106:342-349.
14. Daw N.D. Trial-by-trial data analysis using computational models. Decision making, affect, and learning: Attention and performance XXIII 2011, 3-38. Oxford University Press. M.R. Delgado, E.A. Phelps, T.R. Robbins (Eds.).
15. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
16. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 2005, 8:1704-1711.
17. Daw N., Shohamy D. The cognitive neuroscience of motivation and learning. Social Cognition 2008, 26:593-620.
18. Dayan P. Improving generalization for temporal difference learning: The successor representation. Neural Computation 1993, 5:613-624.
19. Den Ouden H., Daw N.D., Fernandez G., Elshout J., Rijpkema M., Hoogman M., Franke B., Cools R. Dissociable effects of dopamine and serotonin on reversal learning. Neuron 2013, 80(4):1090-1100.
20. Dickinson A. Contemporary animal learning theory 1980, Cambridge University Press.
21. Dickinson A., Smith J., Mirenowicz J. Dissociation of pavlovian and instrumental incentive learning under dopamine antagonists. Behavioral Neuroscience 2000, 114:468-483.
22. Doll B.B., Simon D.A., Daw N.D. The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology 2012, 22:1075-1081.
23. Eichenbaum H., Yonelinas A.P., Ranganath C. The medial temporal lobe and recognition memory. Annual review of Neuroscience 2007, 30:123-152.
24. Foerde K., Knowlton B.J., Poldrack R.A. Modulation of competing memory systems by distraction. Proceedings of the National Academy of Science USA 2006, 103:11778-11783.
25. Gabrieli J.D. Cognitive neuroscience of human memory. Annual Review of Psychology 1998, 49:87-115.
26. Gabrieli J.D., Corkin S., Mickel S.F., Growdon J.H. Intact acquisition and long-term retention of mirror-tracing skill in Alzheimer's disease and in global amnesia. Behavioral Neuroscience 1993, 107:899-910.
27. Gelman A., Carlin J.B., Stern H.S., Rubin D.B. Bayesian data analysis 2003, CRC Press. 2nd ed.
28. Gelman A., Hill J. Data analysis using regression and multilevel/hierarchical models 2007, Cambridge University Press.
29. Gelman A., Rubin D.B. Inference from iterative simulation using multiple sequences. Statistical Science 1992, 7:457-511.
30. Gershman S.J., Markman A.B., Otto A.R. Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General 2012.
31. Glascher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
32. Glimcher P.W. Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proceedings of the National Academy of Science 2011, 108:15647-15654.
33. Greene A.J., Gross W.L., Elsinger C.L., Rao S.M. An FMRI analysis of the human hippocampus: Inference, context, and task awareness. Journal of Cognitive Neuroscience 2006, 18:1156-1173.
34. Hassabis D., Kumaran D., Vann S.D., Maguire E.A. Patients with hippocampal amnesia cannot imagine new experiences. Proceedings of the National Academy of Science USA 2007, 104:1726-1731.
35. Heindel W.C., Salmon D.P., Shults C.W., Walicke P.A., Butters N. Neuropsychological evidence for multiple implicit memory systems: A comparison of Alzheimer's, Huntington's, and Parkinson's disease patients. The Journal of Neuroscience 1989, 9:582-587.
36. Hikida T., Kimura K., Wada N., Funabiki K., Nakanishi S. Distinct roles of synaptic transmission in direct and indirect striatal pathways to reward and aversive behavior. Neuron 2010, 66:896-907.
37. Honey R.C., Hall G. Acquired equivalence and distinctiveness of cues. Journal of Experimental Psychology: Animal Behaviour Processes 1989, 15:338-346.
38. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. The Journal of Neuroscience 2007, 27:12176-12189.
39. Knowlton B.J., Mangels J.A., Squire L.R. A neostriatal habit learning system in humans. Science 1996, 273:1399-1402.
40. Knowlton B.J., Squire L.R., Gluck M.A. Probabilistic classification learning in amnesia. Learning & Memory 1994, 1:106-120.
41. Kruschke J.K. Doing Bayesian data analysis, a tutorial introduction with R 2010, Academic Press.
42. Lee H., Ghim J.W., Kim H., Lee D., Jung M. Hippocampal neural correlates for values of experienced events. The Journal of Neuroscience 2012, 32:15053-15065.
43. Martone M., Butters N., Payne M., Becker J.T., Sax D.S. Dissociations between skill learning and verbal recognition in amnesia and dementia. Archives of Neurology 1984, 41:965-970.
44. Meeter M., Shohamy D., Myers C.E. Acquired equivalence changes stimulus representations. Journal of the Experimental Analysis of Behavior 2009, 91:127-141.
45. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience 1996, 16:1936-1947.
46. Moore A.W., Atkeson C.G. Prioritized sweeping: Reinforcement learning with less data and less time. Machine Learning 1993, 13:103-130.
47. Myers C.E., Shohamy D., Gluck M.A., Grossman S., Kluger A., Ferris S., et al. Dissociating hippocampal versus basal ganglia contributions to learning and transfer. Journal of Cognitive Neuroscience 2003, 15:185-193.
48. Nissen M.J., Bullemer P. Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology 1987, 19:1-32.
49. O'Keefe J., Nadel L. The hippocampus as a cognitive map. Behavioral and Brain Sciences 1979, 2:487-533.
50. Otto A.R., Gershman S.J., Markman A.B., Daw N.D. The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science 2013, 24:751-761.
51. Otto A.R., Raio C.M., Chiang A., Phelps E.A., Daw N.D. Working-memory capacity protects model-based learning from stress. Proceedings of the National Academy of Science USA 2013, 110:20941-20946.
52. Pfeiffer B.E., Foster D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 2013, 497:74-79.
53. Preston A.R., Shrager Y., Dudukovic N.M., Gabrieli J.D.E. Hippocampal contribution to the novel use of relational information in declarative memory. Hippocampus 2004, 14:148-152.
54. Reynolds J.N.J., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks 2002, 15:507-521.
55. Rummery G., Niranjan M. On-line Q-learning using connectionist systems. CUED/F-INFENG/TR 166 1994, Cambridge University Engineering Department.
56. Schapiro A.C., Kustner L.V., Turk-Browne N.B. Shaping of object representations in the human medial temporal lobe based on temporal regularities. Current Biology 2012, 22:1622-1627.
57. Shohamy D., Myers C.E., Grossman S., Sage J., Gluck M.A., Poldrack R.A. Cortico-striatal contributions to feedback-based learning: Converging data from neuroimaging and neuropsychology. Brain 2004, 127:851-859.
58. Shohamy D., Wagner A.D. Integrating memories in the human brain: Hippocampal-midbrain encoding of overlapping events. Neuron 2008, 60:378-389.
59. Simon D.A., Daw N.D. Neural correlates of forward planning in a spatial decision task in humans. The Journal of Neuroscience 2011, 31:5526-5539.
60. Skatova A., Chan P.A., Daw N.D. Extraversion differentiates between model-based and model-free strategies in a reinforcement learning task. Frontiers in Human Neuroscience 2013, 7:525.
61. Squire L.R. Memory and the hippocampus: A synthesis from findings with rats, monkeys, and humans. Psychological Review 1992, 99:195-231.
62. Stan Development Team. Stan: A C++ Library for Probability and Sampling, Version 1.3.
63. Staresina B.P., Davachi L. Mind the gap: Binding experiences across space and time in the human hippocampus. Neuron 2009, 63:267-276.
64. Steinberg E.E., Keiflin R., Boivin J.R., Witten I.B., Deisseroth K., Janak P.H. A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience 2013, 16:966-973.
65. Sutton R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning, GTE Laboratories Incorporated 1990, 216-224. Morgan Kaufmann.
66. Sutton R.S., Barto A.G. Reinforcement learning: An introduction 1998, MIT Press, Cambridge, MA.
67. Tanaka S.C., Doya K., Okada G., Ueda K., Okamoto Y., Yamawaki S. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience 2004, 7:887-893.
68. Tolman E.C. Cognitive maps in rats and men. Psychological Review 1948, 55:189-208.
69. Tolman E.C., Honzik C.H. Introduction and removal of reward, and maze performance in rats. University of California Publications in Psychology 1930, 4:257-275.
70. Tricomi E., Balleine B.W., O'Doherty J.P. A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience 2009, 29:2225-2232.
71. Tsai H.C., Zhang F., Adamantidis A., Stuber G.D., Bonci A., de Lecea L., et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 2009, 324:1080-1084.
72. Turk-Browne N.B., Scholl B.J., Chun M.M., Johnson M.K. Neural evidence of statistical learning: Efficient detection of visual regularities without awareness. Journal of Cognitive Neuroscience 2009, 21:1934-1945.
73. van der Meer M.A.A., Johnson A., Schmitzer-Torbert N.C., Redish A.D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32.
74. Wimmer G.E., Daw N.D., Shohamy D. Generalization of value in reinforcement learning by humans. European Journal of Neuroscience 2012, 35:1092-1104.
75. Wimmer G.E., Shohamy D. Preference by association: How memory mechanisms in the hippocampus bias decisions. Science 2012, 338:270-273.
76. Wunderlich K., Smittenaar P., Dolan R.J. Dopamine enhances model-based over model-free choice behavior. Neuron 2012, 75:418-424.
77. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. European Journal of Neuroscience 2004, 19:181-189.
78. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. European Journal of Neuroscience 2005, 22:513-523.
79. Zeithamova D., Schlichting M.L., Preston A.R. The hippocampus and inferential reasoning: Building memories to navigate future decisions. Frontiers in Human Neuroscience 2012, 6:70.