Internally generated sequences in learning and executing goal-directed behavior

Trends in Cognitive Sciences

Volumn 18, Issue 12, 2014, Pages 647-657

  Pezzulo, Giovanni ^a   van der Meer, Matthijs A A ^b   Lansink, Carien S ^c,d   Pennartz, Cyriel M A ^c,d  

^a INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

^b UNIVERSITY OF WATERLOO   (Canada)

^c UNIVERSITY OF AMSTERDAM   (Netherlands)

^d UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

Decision making; Forward sweep; Generative models; Hippocampus; Inference; Prospection; Reinforcement learning; Replay; Spatial navigation; Theta rhythm; Ventral striatum

Indexed keywords

REINFORCEMENT LEARNING;

FORWARD SWEEPS; GENERATIVE MODEL; HIPPOCAMPUS; INFERENCE; PROSPECTION; REPLAY; SPATIAL NAVIGATION; STRIATUM; THETA RHYTHMS;

DECISION MAKING;

BEHAVIOR; CORPUS STRIATUM; DECISION MAKING; GOAL DIRECTED BEHAVIOR; HIPPOCAMPUS; HUMAN; LEARNING; MEMORY CONSOLIDATION; NERVE CELL; NERVE CELL PLASTICITY; NEUROPHYSIOLOGY; NONHUMAN; PROBABILITY; REVIEW; REWARD; SENSORY STIMULATION; WORKING MEMORY; ANIMAL; BRAIN; HABIT; MEMORY; MOTIVATION; PHYSIOLOGY; PSYCHOMOTOR PERFORMANCE;

ANIMALS; BRAIN; DECISION MAKING; GOALS; HABITS; HUMANS; LEARNING; MEMORY; PSYCHOMOTOR PERFORMANCE;

EID: 84927125372     PISSN: 13646613     EISSN: 1879307X     Source Type: Journal    
DOI: 10.1016/j.tics.2014.06.011     Document Type: Review

References (124)

1. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
2. Tolman E.C. Cognitive maps in rats and men. Psychol. Rev. 1948, 55:189-208.
3. Bornstein A.M., Daw N.D. Multiplicity of control in the basal ganglia: computational roles of striatal subregions. Curr. Opin. Neurobiol. 2011, 21:374-380.
4. Gruber A.J., McDonald R.J. Context, emotion, and the strategic pursuit of goals: interactions among multiple brain systems controlling motivated behavior. Front. Behav. Neurosci 2012, 6:50.
5. Smith K.S., et al. Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proc. Natl. Acad. Sci. U.S.A. 2012, 109:18932-18937.
6. van der Meer M., et al. Information processing in decision-making systems. Neuroscientist 2012, 18:342-359.
7. Niv Y., et al. Choice values. Nat. Neurosci. 2006, 9:987-988.
8. Daw N.D., et al. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
9. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, MIT Press.
10. O'Keefe J., Dostrovsky J. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res. 1971, 34:171-175.
11. O'Keefe J., Conway D.H. Hippocampal place units in the freely moving rat: why they fire where they fire. Exp. Brain Res. 1978, 31:573-590.
12. Wood E.R., et al. The global record of memory in hippocampal neuronal activity. Nature 1999, 397:613-616.
13. Leutgeb S., et al. Independent codes for spatial and episodic memory in hippocampal neuronal ensembles. Science 2005, 309:619-623.
14. Lansink C.S., et al. Reward cues in space: commonalities and differences in neural coding by hippocampal and ventral striatal ensembles. J. Neurosci. 2012, 32:12444-12459.
15. Pastalkova E., et al. Internally generated cell assembly sequences in the rat hippocampus. Science 2008, 321:1322-1327.
16. Kraus B.J., et al. Hippocampal "time cells": time versus path integration. Neuron 2013, 78:1090-1101.
17. Wallis J.D., et al. Single neurons in prefrontal cortex encode abstract rules. Nature 2001, 411:953-956.
18. Durstewitz D., et al. Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning. Neuron 2010, 66:438-448.
19. Mante V., et al. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 2013, 503:78-84.
20. Kehagia A.A., et al. Learning and cognitive flexibility: frontostriatal function and monoaminergic modulation. Curr. Opin. Neurobiol. 2010, 20:199-204.
21. Mulder A.B., et al. Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning. Behav. Brain Res. 2003, 146:77-88.
22. Hok V., et al. Coding for spatial goals in the prelimbic/infralimbic area of the rat frontal cortex. Proc. Natl. Acad. Sci. U.S.A. 2005, 102:4602-4607.
23. Genovesio A., et al. Encoding goals but not abstract magnitude in the primate prefrontal cortex. Neuron 2012, 74:656-662.
24. Parkinson J.A., et al. Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by D-amphetamine. J. Neurosci. 1999, 19:2401-2411.
25. Di Ciano P., et al. Differential involvement of NMDA, AMPA/kainate, and dopamine receptors in the nucleus accumbens core in the acquisition and performance of pavlovian approach behavior. J. Neurosci. 2001, 21:9471-9477.
26. Pennartz C.M.A., et al. The hippocampal-striatal axis in learning, prediction and goal-directed behavior. Trends Neurosci. 2011, 34:548-559.
27. Voorn P., et al. Putting a spin on the dorsal-ventral divide of the striatum. Trends Neurosci. 2004, 27:468-474.
28. Mendelsohn A., et al. Between thoughts and actions: motivationally salient cues invigorate mental action in the human brain. Neuron 2014, 81:207-217.
29. Roesch M.R., et al. Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J. Neurosci. 2009, 29:13365-13376.
30. Wilson M.A., McNaughton B.L. Reactivation of hippocampal ensemble memories during sleep. Science 1994, 265:676-679.
31. Diba K., Buzsáki G. Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 2007, 10:1241-1242.
32. Foster D.J., Wilson M.A. Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 2006, 440:680-683.
33. Carr M.F., et al. Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nat. Neurosci. 2011, 14:147-153.
34. Johnson A., Redish A.D. Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci. 2007, 27:12176-12189.
35. van der Meer M.A.A., et al. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32.
36. Pfeiffer B.E., Foster D.J. Hippocampal place-cell sequences depict future paths to remembered goals. Nature 2013, 497:74-79.
37. Dragoi G., Tonegawa S. Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 2011, 469:397-401.
38. Chersi F., et al. Mental imagery in the navigation domain: a computational model of sensory-motor simulation mechanisms. Adaptive Behav. 2013, 21:251-262.
39. Erdem U.M., Hasselmo M. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. Eur. J. Neurosci. 2012, 35:916-931.
40. Penny W.D., et al. Forward and backward inference in spatial cognition. PLoS Comput. Biol. 2013, 9:e1003383.
41. Mittelstaedt M.L., Mittelstaedt H. Homing by path integration in a mammal. Naturwissenschaften 1980, 67:566-567.
42. McNaughton B.L., et al. Path integration and the neural basis of the 'cognitive map'. Nat. Rev. Neurosci. 2006, 7:663-678.
43. Lansink C.S., et al. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 2009, 7:e1000173.
44. Pezzulo G., et al. The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front. Psychol. 2013, 4:92.
45. Yin H.H., et al. Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur. J. Neurosci. 2008, 28:1437-1448.
46. Balleine B.W., et al. The integrative function of the basal ganglia in instrumental conditioning. Behav. Brain Res. 2009, 199:43-52.
47. Jadhav S.P., et al. Awake hippocampal sharp-wave ripples support spatial memory. Science 2012, 336:1454-1458.
48. Gupta A.S., et al. Hippocampal replay is not a simple function of experience. Neuron 2010, 65:695-705.
49. Botvinick M., Toussaint M. Planning as inference. Trends Cogn. Sci. 2012, 16:485-488.
50. Wagner U., et al. Sleep inspires insight. Nature 2004, 427:352-355.
51. Friston K., et al. Active inference and agency: optimal control without cost functions. Biol. Cybern. 2012, 106:523-541.
52. Kearns M., et al. A sparse sampling algoritm for near-optimal planning in a large Markov decision process. Machine Learn. 2002, 49:193-208.
53. Lansink C.S., et al. Preferential reactivation of motivationally relevant information in the ventral striatum. J. Neurosci. 2008, 28:6372-6382.
54. van der Meer M.A.A., Redish A.D. Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci. 2009, 3:1.
55. Solway A., Botvinick M.M. Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychol. Rev. 2012, 119:120-154.
56. Singer A.C., et al. Hippocampal SWR activity predicts correct decisions during the initial learning of an alternation task. Neuron 2013, 77:1163-1173.
57. Davidson T.J., et al. Hippocampal replay of extended experience. Neuron 2009, 63:497-507.
58. Kjelstrup K.B., et al. Finite scale of spatial representation in the hippocampus. Science 2008, 321:140-143.
59. Hyman J.M., et al. Contextual encoding by ensembles of medial prefrontal cortex neurons. Proc. Natl. Acad. Sci. U.S.A. 2012, 109:5086-5091.
60. Silver D., Veness J. Monte-Carlo Planning in Large POMDPs. NIPS 2010, 2164-2172. Curran Associates. J.D. Lafferty (Ed.).
61. Geffner H., Bonet B. A concise introduction to models and methods for automated planning. Synthesis Lect. Artif. Intell. Machine Learn. 2013, 8:1-141.
62. Battaglia F.P., Pennartz C.M. The construction of semantic memory: grammar-based representations learned from relational episodic information. Front. Comput. Neurosci. 2011, 5:36.
63. Ratcliff R. A theory of memory retrieval. Psychol. Rev. 1978, 85:59-108.
64. Stewart N., et al. Decision by sampling. Cognit. Psychol. 2006, 53:1-26.
65. Shadlen M.N., Kiani R. Decision making as a window on cognition. Neuron 2013, 80:791-806.
66. van Dongen Y.C., et al. Anatomical evidence for direct connections between the shell and core subregions of the rat nucleus accumbens. Neuroscience 2005, 136:1049-1071.
67. Leblois A., et al. Competition between feedback loops underlies normal and pathological dynamics in the basal ganglia. J. Neurosci. 2006, 26:3567-3583.
68. Redgrave P., et al. Functional properties of the basal ganglia's re-entrant loop architecture: selection and reinforcement. Neuroscience 2011, 198:138-151.
69. Zhang J., et al. A neural computational model of incentive salience. PLoS Comput. Biol. 2009, 5:e1000437.
70. Dalley J.W., et al. Time-limited modulation of appetitive Pavlovian memory by D1 and NMDA receptors in the nucleus accumbens. Proc. Natl. Acad. Sci. U.S.A. 2005, 102:6189-6194.
71. Baldo B.A., Kelley A.E. Discrete neurochemical coding of distinguishable motivational processes: insights from nucleus accumbens control of feeding. Psychopharmacology (Berl.) 2007, 191:439-459.
72. Pennartz C.M., et al. Corticostriatal Interactions during learning, memory processing, and decision making. J. Neurosci. 2009, 29:12831-12838.
73. Haber S.N., et al. Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci. 2000, 20:2369-2382.
74. Belin D., Everitt B.J. Cocaine seeking habits depend upon dopamine-dependent serial connectivity linking the ventral with the dorsal striatum. Neuron 2008, 57:432-441.
75. Friston K. The free-energy principle: a unified brain theory?. Nat. Rev. Neurosci. 2010, 11:127-138.
76. Dolan R.J., Dayan P. Goals and habits in the brain. Neuron 2013, 80:312-325.
77. Gershman S.J., et al. Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. 2014, 143:182-194.
78. Sutton R.S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning 1990, 216-224.
79. Tulving E. Elements of episodic memory 1983, Clarendon Press.
80. Huijbers W., et al. Imagery and retrieval of auditory and visual information: Neural correlates of successful and unsuccessful performance. Neuropsychologia 2011, 49:1730-1740.
81. Hassabis D., et al. Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci. U.S.A. 2007, 104:1726-1731.
82. Raichle M.E., et al. A default mode of brain function. Proc. Natl. Acad. Sci. U.S.A. 2001, 98:676-682.
83. Daselaar S.M., et al. Modality-specific and modality-independent components of the human imagery system. Neuroimage 2010, 52:677-685.
84. Andrews-Hanna J.R., et al. Functional-anatomic fractionation of the brain's default network. Neuron 2010, 65:550-562.
85. Szpunar K.K., et al. Neural substrates of envisioning the future. Proc. Natl. Acad. Sci. U.S.A. 2007, 104:642-647.
86. Addis D.R., et al. Hippocampal contributions to the episodic simulation of specific and general future events. Hippocampus 2011, 21:1045-1052.
87. Mason M.F., et al. Wandering minds: the default network and stimulus-independent thought. Science 2007, 315:393-395.
88. Ostby Y., et al. Mental time travel and default-mode network functional connectivity in the developing brain. Proc. Natl. Acad. Sci. U.S.A. 2012, 109:16800-16804.
89. Friston K., et al. Action understanding and active inference. Biol. Cybern. 2011, 104:137-160.
90. Yamashita Y., Tani J. Emergence of functional hierarchy in a multiple timescale neural network model: a humanoid robot experiment. PLoS Comput. Biol. 2008, 4:e1000220.
91. Rabinovich M.I., et al. Transient cognitive dynamics, metastability, and decision making. PLoS Comput. Biol. 2008, 4:e1000072.
92. Murphy K. Machine learning: a probabilistic perspective 2012, MIT Press.
93. Bendor D., Wilson M.A. Biasing the content of hippocampal replay during sleep. Nat. Neurosci. 2012, 15:1439-1444.
94. Keramati M., et al. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 2011, 7:e1002055.
95. Gold J.I., Shadlen M.N. The neural basis of decision making. Annu. Rev. Neurosci. 2007, 30:535-574.
96. Nakahara H., Hikosaka O. Learning to represent reward structure: a key to adapting to complex environments. Neurosci. Res. 2012, 74:177-183.
97. Johnson A., Redish A.D. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw. 2005, 18:1163-1171.
98. Connor P.C., et al. An elemental model of retrospective revaluation without within-compound associations. Learn. Behav. 2013, 42:22-38.
99. Kudrimoti H.S., et al. Reactivation of hippocampal cell assemblies: effects of behavioral state, experience, and EEG dynamics. J. Neurosci. 1999, 19:4090-4101.
100. Pennartz C.M.A., et al. The ventral striatum in off-line processing: ensemble reactivation during sleep and modulation by hippocampal ripples. J. Neurosci. 2004, 24:6446-6456.
101. Buzsáki G. Hippocampal sharp waves: their origin and significance. Brain Res. 1986, 398:242-252.
102. Euston D.R., et al. Fast-forward playback of recent memory sequences in prefrontal cortex during sleep. Science 2007, 318:1147-1150.
103. Skaggs W.E., McNaughton B.L. Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience. Science 1996, 271:1870-1873.
104. Nádasdy Z., et al. Replay and time compression of recurring spike sequences in the hippocampus. J. Neurosci. 1999, 19:9497-9507.
105. Abbott L.F., Nelson S.B. Synaptic plasticity: taming the beast. Nat. Neurosci. 2000, 3(Suppl):1178-1183.
106. Markram H., et al. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science 1997, 275:213-215.
107. Lee A.K., Wilson M.A. Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 2002, 36:1183-1194.
108. Girardeau G., et al. Selective suppression of hippocampal ripples impairs spatial memory. Nat. Neurosci. 2009, 12:1222-1223.
109. Ego-Stengel V., Wilson M.A. Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat. Hippocampus 2010, 20:1-10.
110. Qin Y.L., et al. Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 1997, 352:1525-1533.
111. Ji D., Wilson M.A. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nat. Neurosci. 2007, 10:100-107.
112. Karlsson M.P., Frank L.M. Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 2009, 12:913-918.
113. Dragoi G., Tonegawa S. Distinct preplay of multiple novel spatial experiences in the rat. Proc. Natl. Acad. Sci. U.S.A. 2013, 110:9100-9105.
114. Kappen H.J., et al. Optimal control as a graphical model inference problem. Machine Learn. 2009, 87:159-182.
115. Ortega P.A., Braun D.A. Thermodynamics as a theory of decision-making with information-processing costs. Proc. R. Soc. 2013, 469:20120683.
116. Kepecs A., et al. Neural correlates, computation and behavioural impact of decision confidence. Nature 2008, 455:227-231.
117. Dayan P. Twenty-five lessons from computational neuromodulation. Neuron 2012, 76:240-256.
118. Schultz W., et al. Explicit neural signals reflecting reward uncertainty. Philos. Trans. R. Soc. Lond. B: Biol. Sci. 2008, 363:3801-3811.
119. Cabral H.O., et al. Oscillatory dynamics and place field maps reflect hippocampal ensemble processing of sequence and place memory under NMDA receptor control. Neuron 2014, 81:402-415.
120. Day J.J., et al. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 2007, 10:1020-1028.
121. Liljeholm M., O'Doherty J.P. contributions of the striatum to learning, motivation, and performance: An associative account. Trends Cogn. Sci. 2012, 16:1364-6613.
122. Everitt B.J., Robbins T.W. Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci. 2005, 8:1481-1489.
123. de Borchgrave R., et al. Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp. Brain Res. 2002, 144:50-68.
124. Miller J.F., et al. Neural activity in human hippocampal formation reveals the spatial context of retrieved memories. Science 2013, 342:1111-1114.