Integrating memories to guide decisions

Current Opinion in Behavioral Sciences

Volumn 5, Issue , 2015, Pages 85-90

  Shohamy, Daphna ^a   Daw, Nathaniel D ^b  

^a Department of Psychology Columbia University ^*  (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANTEROGRADE AMNESIA; ASSOCIATION; DECISION MAKING; DELAY DISCOUNTING; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HIPPOCAMPUS; HUMAN; INSTRUMENTAL CONDITIONING; LOCOMOTION; MEDIAL TEMPORAL LOBE; MEMORY; NEUROIMAGING; NONHUMAN; PREFRONTAL CORTEX; REINFORCEMENT; REVIEW; REWARD; SPATIAL ORIENTATION; STATE DEPENDENT LEARNING;

EID: 84941026115     PISSN: None     EISSN: 23521546     Source Type: Journal    
DOI: 10.1016/j.cobeha.2015.08.010     Document Type: Review

References (61)

1. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of information processing in the basal ganglia 1995, 249-270.
2. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
3. Dickinson A., Balleine B. The role of learning in the operation of motivational systems. Stevens' handbook of experimental psychology 2002.
4. Tolman E.C. Cognitive maps in rats and men. Psychol Rev 1948, 55:189-208.
5. Daw N.D., Dayan P. The algorithmic anatomy of model-based evaluation. Philos Trans R Soc Lond B Biol Sci 2014, 369.
6. Daw N.D., O'Doherty J.P. Multiple systems for value learning. Neuroeconomics, decision making and the brain 2014, 393-410. Academic Press, Amsterdam. 2nd ed. P. Glimcher, E. Fehr (Eds.).
7. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
8. Balleine B.W., O'Doherty J.P. Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 2010, 35:48-69.
9. Camerer C., Hua Ho T. Experience-weighted attraction learning in normal form games. Econometrica 1999, 67:827-874.
10. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
11. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J Neurosci 2011, 31:2700-2705.
12. Barron H.C., Dolan R.J., Behrens T.E. Online evaluation of novel choices by simultaneous representation of multiple memories. Nat Neurosci 2013, 16:1492-1498.
13. Bornstein A.M., Daw N.D. Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans. PLoS Comput Biol 2013, 9:e1003387.
14. Greene A.J., Gross W.L., Elsinger C.L., Rao S.M. An FMRI analysis of the human hippocampus: inference, context, and task awareness. J Cogn Neurosci 2006, 18:1156-1173.
15. Jones J.L., Esber G.R., McDannald M.A., Gruber A.J., Hernandez A., Mirenzi A., Schoenbaum G. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 2012, 338:953-956.
16. Preston A.R., Shrager Y., Dudukovic N.M., Gabrieli J.D. Hippocampal contribution to the novel use of relational information in declarative memory. Hippocampus 2004, 14:148-152.
17. Gremel C.M., Costa R.M. Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nat Commun 2013, 4:2264.
18. West E.A., DesJardin J.T., Gale K., Malkova L. Transient inactivation of orbitofrontal cortex blocks reinforcer devaluation in macaques. J Neurosci 2011, 31:15128-15135.
19. Stalnaker T.A., Cooch N.K., Schoenbaum G. What the orbitofrontal cortex does not do. Nat Neurosci 2015, 18:620-627.
20. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 2007, 27:12176-12189.
21. Pfeiffer B.E., Foster D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 2013, 497:74-79.
22. Gupta A.S., van der Meer M.A., Touretzky D.S., Redish A.D. Hippocampal replay is not a simple function of experience. Neuron 2010, 65:695-705.
23. Wikenheiser A.M., Redish A.D. Hippocampal theta sequences reflect current goals. Nat Neurosci 2015, 18:289-294.
24. Schapiro A.C., Kustner L.V., Turk-Browne N.B. Shaping of object representations in the human medial temporal lobe based on temporal regularities. Curr Biol 2012, 22:1622-1627.
25. Schapiro A.C., Rogers T.T., Cordova N.I., Turk-Browne N.B., Botvinick M.M. Neural representations of events arise from temporal community structure. Nat Neurosci 2013, 16:486-492.
26. Jadhav S.P., Kemere C., German P.W., Frank L.M. Awake hippocampal sharp-wave ripples support spatial memory. Science 2012, 336:1454-1458.
27. Singer A.C., Carr M.F., Karlsson M.P., Frank L.M. Hippocampal SWR activity predicts correct decisions during the initial learning of an alternation task. Neuron 2013, 77:1163-1173.
28. Doll B.B., Duncan K.D., Simon D.A., Shohamy D., Daw N.D. Model-based choices involve prospective neural activity. Nat Neurosci 2015, 18:767-772.
29. Hassabis D., Kumaran D., Vann S.D., Maguire E.A. Patients with hippocampal amnesia cannot imagine new experiences. Proc Natl Acad Sci U S A 2007, 104:1726-1731.
30. Wheeler M.A., Stuss D.T., Tulving E. Toward a theory of episodic memory: the frontal lobes and autonoetic consciousness. Psychol Bull 1997, 121:331-354.
31. Addis D.R., Wong A.T., Schacter D.L. Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia 2007, 45:1363-1377.
32. Summerfield J.J., Hassabis D., Maguire E.A. Differential engagement of brain regions within a 'core' network during scene construction. Neuropsychologia 2010, 48:1501-1509.
33. Peters J., Buchel C. Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron 2010, 66:138-148.
34. Palombo D.J., Keane M.M., Verfaellie M. The medial temporal lobes are critical for reward-based decision making under conditions that promote episodic future thinking. Hippocampus 2015, 25(3):345-353.
35. Foster D.J., Wilson M.A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 2006, 440:680-683.
36. Louie K., Wilson M.A. Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron 2001, 29:145-156.
37. Karlsson M.P., Frank L.M. Awake replay of remote experiences in the hippocampus. Nat Neurosci 2009, 12:913-918.
38. Wu X., Foster D.J. Hippocampal replay captures the unique topological structure of a novel environment. J Neurosci 2014, 34:6459-6469.
39. Olafsdottir H.F., Barry C., Saleem A.B., Hassabis D., Spiers H.J. Hippocampal place cells construct reward related sequences through unexplored space. Elife 2015, 4.
40. Kurth-Nelson Z., Barnes G., Sejdinovic D., Dolan R., Dayan P. Temporal structure in associative retrieval. eLife 2015, 4:e04919.
41. Wimmer G.E., Shohamy D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 2012, 338:270-273.
42. Kumaran D., Summerfield J.J., Hassabis D., Maguire E.A. Tracking the emergence of conceptual knowledge during human decision making. Neuron 2009, 63:889-901.
43. Shohamy D., Wagner A.D. Integrating memories in the human brain: hippocampal-midbrain encoding of overlapping events. Neuron 2008, 60:378-389.
44. Zeithamova D., Dominick A.L., Preston A.R. Hippocampal and ventral medial prefrontal activation during retrieval-mediated learning supports novel inference. Neuron 2012, 75:168-179.
45. Sutton R.S. Dyna an integrated architecture for learning, planning, and reacting. ACM SIGART Bull 1991, 2:160-163.
46. Johnson A., Redish A.D. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw 2005, 18:1163-1171.
47. Li J., Schiller D., Schoenbaum G., Phelps E.A., Daw N.D. Differential roles of human striatum and amygdala in associative learning. Nat Neurosci 2011, 14:1250-1252.
48. Deserno L., Huys Q.J., Boehme R., Buchert R., Heinze H.J., Grace A.A., Dolan R.J., Heinz A., Schlagenhauf F. Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proc Natl Acad Sci U S A 2015, 112:1595-1600.
49. Wunderlich K., Smittenaar P., Dolan R.J. Dopamine enhances model-based over model-free choice behavior. Neuron 2012, 75:418-424.
50. Gershman S.J., Markman A.B., Otto A.R. Retrospective revaluation in sequential decision making: a tale of two systems. J Exp Psychol Gen 2014, 143:182-194.
51. Ellenbogen J.M., Hu P.T., Payne J.D., Titone D., Walker M.P. Human relational memory requires time and sleep. Proc Natl Acad Sci U S A 2007, 104:7723-7728.
52. Keramati M., Dezfouli A., Piray P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 2011, 7:e1002055.
53. Pezzulo G., Rigoli F., Chersi F. The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front Psychol 2013, 4.
54. Moore A.W., Atkeson C.G. Prioritized sweeping-reinforcement learning with less data and less time. Mach Learn 1993, 13:103-130.
55. Kurzban R., Duckworth A., Kable J.W., Myers J. An opportunity cost model of subjective effort and task performance. Behav Brain Sci 2013, 36:661-679.
56. Shenhav A., Botvinick M.M., Cohen J.D. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 2013, 79:217-240.
57. McClelland J.L., McNaughton B.L., O'Reilly R.C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychol Rev 1995, 102:419-457.
58. Biele G., Erev I., Ert E. Learning, risk attitude and hot stoves in restless bandit problems. J Math Psychol 2009, 53:155-167.
59. Collins A.G., Frank M.J. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci 2012, 35:1024-1035.
60. Cohen N.J., Eichenbaum H. Memory A: the hippocampal system 1993, MIT Press, Cambridge, MA.
61. Lengyel M., Dayan P. Hippocampal contributions to control: the third way. NIPS 2007, 889-896.