Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model

Neural Networks

Volumn 18, Issue 9, 2005, Pages 1163-1171

  Johnson, Adam ^a   Redish, A David ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; COMPUTATION THEORY; DISEASES; MATHEMATICAL MODELS; NEURAL NETWORKS;

GANGLIA; HIPPOCAMPAL INACTIVATION; HIPPOCAMPAL REPLAY; TEMPORAL DIFFERENCE REINFORCEMENT LEARNING (TDRL);

LEARNING SYSTEMS;

ALGORITHM; ARTICLE; BRAIN NERVE CELL; HIPPOCAMPUS; HYPOTHESIS; LEARNING; MODEL; PRIORITY JOURNAL; REINFORCEMENT; REST; SLEEP; TASK PERFORMANCE; VELOCITY;

ALGORITHMS; ANIMALS; BEHAVIOR, ANIMAL; COMPUTER SIMULATION; HIPPOCAMPUS; MAZE LEARNING; MEMORY; MODELS, NEUROLOGICAL; RATS; REINFORCEMENT (PSYCHOLOGY); REWARD; SPACE PERCEPTION;

EID: 27844567151     PISSN: 08936080     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.neunet.2005.08.009     Document Type: Article

References (87)

1. G.A. Ascoli, and V. Samsonovich Connectionist model of the hippocampus suggesting a new link between episodic memory and spatial navigation Society for Neuroscience Abstracts, Program number 2004 66714
2. A.G. Barto Adaptive critics and the basal ganglia J.C. Houk J.L. Davis D.G. Beiser Models of Information Processing in the Basal Ganglia 1995 MIT Press Cambridge MA 215 232
3. G. Bi, and M. Poo Synaptic modification by correlated activity: Hebb's postulate revisited Annual Review of Neuroscience 24 1 2001 139 166
4. K.I. Blum, and F. Abbott A model of spatial map formation in the hippocampus of the rat Neural Computation 8 1 1996 85 93
5. M.A. Brown, and E. Sharp Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal formation and nucleus accumbens Hippocampus 5 3 1995 171 188
6. G. Buzsáki Two-stage model of memory trace formation: A role for "noisy" brain states Neuroscience 31 3 1989 551 570
7. G. Buzsáki, L.W. Leung, and H. Vanderwolf Cellular bases of hippocampal EEG in the behaving rat Brain Research 287 2 1983 139 171
8. N.J. Cohen, and H. Eichenbaum Memory, Amnesia, and the Hippocampal System 1993 MIT Press Cambridge, MA
9. J. Csicsvari, H. Hirase, A. Czurkó, and G. Buzsaki Fast network oscillations in the hippocampal CA1 region of the behaving rat Journal of Neuroscience 19 RC20 1999 1 4
10. J. Csicsvari, H. Hirase, A. Czurkó, and G. Buzsaki Oscillatory coupling of hippocampal pyramidal cells and interneurons in the behaving rat Journal of Neuroscience 1 1999 274 287
11. J. Csicsvari, H. Hirase, A. Mamiya, and G. Buzsaki Ensemble patterns of hippocampal CA3-CA1 neurons during sharp wave-associated population events Neuron 28 2000 585 594
12. N. D. Daw. Reinforcement learning models of the dopamine system and their behavioral implications. PhD thesis, Carnegie Mellon University, 2003.
13. P. Dayan, and W. Balleine Reward, motivation, and reinforcement learning Neuron 36 2002 285 298
14. A. Dickinson Actions and habits: the development of behavioural autonomy Philosophical Transactions of the Royal Society, London B 308 1985 67 78
15. S. Doboli, A.A. Minai, and J. Best Latent attractors: a model for context-dependent place representations in the hippocampus Neural Computation 12 5 2000 1009 1043
16. K. Doya What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex? Neural networks 12 1999 961 974
17. H. Eichenbaum, M. Kuperstein, and A. Fagan Cue-sampling and goal-approach correlates of hippocampal unit activity in rats performing an odor-discrimination task Journal of Neuroscience 7 3 1987 716 732
18. A.D. Ekstrom, J. Meltzer, B.L. McNaughton, and A. Barnes NMDA receptor antagonism blocks experience-dependent expansion of hippocampal "place fields" Neuron 31 2001 631 638
19. J. Ferbinteanu, and M.L. Shapiro Prospective and retrospective memory coding in the hippocampus Neuron 40 6 2003 1227 1239
20. D.J. Foster, and R.G.M. Morris A model of hippocampally dependent navigation using the temporal difference learning rule Hippocampus 10 2000 1 6
21. A.M. Graybiel, T. Aosaki, and A.W. Flaherty The basal ganglia and adaptive motor control Science 265 5180 1994 1826 1831
22. K.D. Harris, J. Csicsvari, H. Hirase, G. Dragoi, and G. Buzsaki Organization of cell assemblies in the hippocampus Nature 424 6948 2003 552 556
23. L. Heimer, and G.F. Alheid Basal ganglia G. Paxinos The Rat Nervous System 1st ed. 1985 Academic Press Orlando FL 37 86
24. O. Hikosaka, H. Nakahara, M.K. Rand, K. Sakai, X. Lu, K. Nakamura, and S. Miyachi Parallel neural networks for learning sequential procedures Trends in Neurosciences 22 10 1999 464 471
25. K.L. Hoffman, and L. McNaughton Coordinated Reactivation of Distributed Memory Traces in Primate Neocortex Science 297 5589 2002 2070 2073
26. R. Ito, J.W. Dalley, S.R. Howes, T.W. Robbins, and J. Everitt Dissociation in conditioned dopamine release in the nucleus accumbens core and shell in response to cocaine cues and during cocaine-seeking behavior in rats Journal of Neuroscience 20 19 2000 7489 7495
27. R. Ito, J.W. Dalley, T.W. Robbins, and J. Everitt Dopamine release in the dorsal striatum during cocaine-seeking behavior under the control of a drug-associated cue Journal of Neuroscience 22 14 2002 6247 6253
28. H. Itoh, H. Nakahara, O. Hikosaka, R. Kawagoe, and Y. Takikawa Correlation of primate caudate neural activity and saccade parameters in reward-oriented behavior Journal of Neurophysiology 89 4 2003 1774 1783
29. M.S. Jog, Y. Kubota, C.I. Connolly, V. Hillegaart, and M. Graybiel Building neural representations of habits Science 286 1999 1746 1749
30. Clifford G. Kentros, Naveen T. Agnihotri, Samantha Streater, Robert D. Hawkins, and Eric R. Kandel Increased attention to spatial context increases both place field stability and spatial memory Neuron 42 2004 283 295
31. I. Kermadi, Y. Jurquet, M. Arzi, and P. Joseph Neural activity in the caudate nucleus of monkeys during spatial sequencing Experimental Brain Research 94 1993 352 356
32. M. Kimura The role of primate putamen neurons in the association of sensory stimuli with movement Neuroscience Research 3 5 1986 436 443
33. H.S. Kudrimoti, C.A. Barnes, and L. McNaughton Reactivation of hippocampal cell assemblies: Effects of behavioral state, experience, and EEG dynamics Journal of Neuroscience 19 10 1999 4090 4101
34. A.K. Lee, and M.A. Wilson Memory of sequential experience in the hippocampus during slow wave sleep Neuron 36 2002 1183 1194
35. I. Lee, G. Rao, and J. Knierim A double dissociation between hippocampal subfields: Differential time course of ca3 and ca1 place cells for processing changed environments Neuron 42 2004 803 815
36. W.B. Levy A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal-like tasks Hippocampus 6 6 1996 579 591
37. W.B. Levy, and O. Steward Temporal contiguity requirements for long-term associative potentiation/depression in the hippocampus Neuroscience 8 4 1983 791 797
38. K. Louie, and M.A. Wilson Temporally structured replay of awake hippocampal ensemble activity during Rapid Eye Movement sleep Neuron 29 2001 145 156
39. D. Marr Simple memory: A theory of archicortex Philosophical Transactions of the Royal Society of London 262 841 1971 23 81
40. A.J. McGeorge, and L. Faull The organization of the projection from the cerebral cortex to the striatum in the rat Neuroscience 29 3 1989 503 537
41. B.L. McNaughton Associative properties of hippocampal long term potentiation W. Seifert Neurobiology of the Hippocampus 1983 Academic Press New York, NY 433 447
42. M.R. Mehta, C.A. Barnes, and L. McNaughton Experience-dependent, asymmetric expansion of hippocampal place fields Proceedings of the National Academy of Sciences, USA 94 1997 8918 8921
43. M.R. Mehta, M.C. Quirk, and A. Wilson Experience-dependent asymmetric shape of hippocampal receptive fields Neuron 25 2000 707 715
44. G.J. Mogenson, D.L. Jones, and Y. Yim From motivation to action: Functional interface between the limbic system and the motor system Progress in Neurobiology 14 1980 69 97
45. M.A. Moita, S. Rosis, Y. Zhou, J.E. LeDoux, and T. Blair Hippocampal place cells acquire location-specific responses to the conditioned stimulus during auditory fear conditioning Neuron 37 3 2003 485 497
46. P.R. Montague, P. Dayan, and J. Sejnowski A framework for mesencephalic dopamine systems based on predictive hebbian learning Journal of Neuroscience 16 5 1996 1936 1947
47. Z. Nádasdy, H. Hirase, A. Czurkó, J. Csicsvari, and G. Buzsáki Replay and time compression of recurring spike sequences in the hippocampus Journal of Neuroscience 19 2 1999 9497 9507
48. J.P. O'Doherty, P. Dayan, K. Friston, H. Critchley, and R.J. Dolan Temporal difference models and reward-related learning in the human brain Neuron 38 2 2003 329 337
49. J. O'Keefe, and J. Dostrovsky The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely moving rat Brain Research 34 1971 171 175
50. J. O'Keefe, and L. Nadel The Hippocampus as a Cognitive Map 1978 Clarendon Press Oxford
51. J. O'Keefe, and M. Recce Phase relationship between hippocampal place units and the EEG theta rhythm Hippocampus 3 1993 317 330
52. N.A. Otmakhova, and J.E. Lisman Dopamine selectively inhibits the direct cortical pathway to the CA1 hippocampal region Journal of Neuroscience 19 4 1999 1437 1445
53. N.A. Otmakhova, and J.E. Lisman D1/D5 Dopamine Receptor Activation Increases the Magnitude of Early Long-Term Potentiation at CA1 Hippocampal Synapses Journal of Neuroscience 16 23 1996 7478 7486
54. T. Otto, and H. Eichenbaum Neuronal activity in the hippocampus during delayed non-match to sample performance in rats: Evidence for hippocampal processing in recognition memory Hippocampus 2 3 1992 323 334
55. M.G. Packard, and J. Knowlton Learning and memory functions of the basal ganglia Annual Reviews Neuroscience 25 1 2002 563 593
56. M.G. Packard, and L. McGaugh Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning Neurobiology of Learning and Memory 65 1996 65 72
57. C. Pavlides, and J. Winson Influences of hippocampal place cell firing in the awake state on the activity of these cells during subsequent sleep episodes Journal of Neuroscience 9 8 1989 2907 2918
58. C.M.A. Pennartz, E. Lee, J. Verheul, P. Lipa, C.A. Barnes, and L. McNaughton The ventral striatum in off-line processing: ensemble reactivation during sleep and modulation by hippocampal ripples Journal of Neuroscience 24 29 2004 6446 6456
59. R.A. Poldrack, and G. Packard Competition among multiple memory systems: Converging evidence from animal and human studies Neuropsychologia 41 2003 245 251
60. Y.L. Qin, B.L. McNaughton, E. Skaggs, and A. Barnes Memory reprocessing in corticocortical and hippocampocortical neuronal ensembles Philosophical Transactions of the Royal Society, London B352 1360 1997 1525 1533
61. A.D. Redish Beyond the Cognitive Map: From Place Cells to Episodic Memory 1999 MIT Press Cambridge MA
62. A.D. Redish, A. Johnson, and S. Jensen Latent learning requires multiple value functions with temporal difference reinforcement learning Computational Neurosciences Abstracts 2005
63. A.D. Redish, and S. Touretzky Cognitive maps beyond the hippocampus Hippocampus 7 1 1997 15 35
64. A.D. Redish, and S. Touretzky The role of the hippocampus in solving the Morris water maze Neural Computation 10 1 1998 73 111
65. F. Restle Discrimination of cues in mazes: A resolution of the 'place-vs-response' question Psychological Review 64 1957 217 228
66. Y. Sakurai Hippocampal cells have behavioral correlates during the performance of an auditory working memory task in the rat Behavioral Neuroscience 104 2 1990 253 263
67. A.V. Samsonovich, and B.L. McNaughton Path integration and cognitive mapping in a continuous attractor neural network model Journal of Neuroscience 17 15 1997 5900 5920
68. N.C. Schmitzer-Torbert, and A.D. Redish Development of path stereotypy in a single day in rats on a multiple-T maze Archives Italiennes de Biologie 140 2002 295 301
69. N.C. Schmitzer-Torbert, and A.D. Redish Neuronal activity in the rodent dorsal striatum in sequential navigation: Separation of spatial and reward responses on the multiple-T task Journal of Neurophysiology 91 5 2004 2259 2272
70. W. Schultz Predictive reward signal of dopamine neurons Journal of Neurophysiology 80 1998 1 27
71. W. Schultz, and P. Dayan A neural substrate of prediction and reward Science 275 1997 1593 1599
72. B. Seymour, J.P. O'Doherty, P. Dayan, M. Koltzenburg, A.K. Jones, R.J. Dolan, K.J. Friston, and S. Frackowlak Temporal difference models describe higher-order learning in humans Nature 429 2004 664 667
73. B. Shen, and B.L. McNaughton Modeling the spontaneous reactivation of experience-specific hippocampal cell assembles during sleep Hippocampus 6 6 1996 685 693
74. J. Shen, C.A. Barnes, B.L. McNaughton, W.E. Skaggs, and L. Weaver The effect of aging on experience-dependent plasticity of hippocampal place cells Journal of Neuroscience 17 17 1997 6769 6782
75. W.E. Skaggs, and B.L. McNaughton Replay of neuronal firing sequences in rat hippocampus during sleep following spatial experience Science 271 1996 1870 1873
76. W.E. Skaggs, B.L. McNaughton, M.A. Wilson, and A. Barnes Theta phase precession in hippocampal neuronal populations and the compression of temporal sequences Hippocampus 6 2 1996 149 173
77. R.S. Sutton Integrated architectures for learning, planning, and reacting based on approximating dynamic programming In Proceedings of the Seventh International Conference on Machine Learning 1990 216 224
78. R.S. Sutton, and A.G. Barto Reinforcement Learning: An introduction 1998 MIT Press Cambridge MA
79. S.C. Tanaka, K. Doya, G. Okada, K. Ueda, Y. Okamoto, and S. Yamawaki Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops Nature Neuroscience 7 2004 887 893
80. E.C. Tolman, B.F. Ritchie, and D. Kalish Studies in spatial learning. II. Place learning versus response learning Journal of Experimental Psychology 36 1946 221 229
81. M. Tsodyks Attractor network models of spatial maps in hippocampus Hippocampus 9 4 1999 481 489
82. P. Waelti, A. Dickinson, and W. Schultz Dopamine responses comply with basic assumptions of formal learning theory Nature 2001 10.1038/35083500
83. M.A. Wilson, and L. McNaughton Reactivation of hippocampal ensemble memories during sleep Science 265 1994 676 679
84. M.P. Witter, and H.J. Groenewegen The subiculum: Cytoarchitectonically a simple structure but hodologically complex J. Storm-Mathisen J. Zimmer O.P Ottersen Understanding the Brain through the Hippocampus Progress in Brain Research vol. 83 1990 Elsevier New York
85. E.R. Wood, P.A. Dudchenko, and R.J. Robitsek Hippocampal neurons encode information about different types of memory episodes occurring in the same location Neuron 27 2000 623 633
86. H.H. Yin, and B.J. Knowlton Contributions of Striatal Subregions to Place and Response Learning Learn. Mem. 11 4 2004 459 463
87. A. Ylinen, A. Bragin, Z. Nadasdy, G. Jando, and G. Buzsaki Sharp wave-associated high-frequency oscillation (200 Hz) in the intact hippocampus: Network and intracellular mechanisms Journal of Neuroscience 15 1 1995 30 46