Grid cells, place cells, and geodesic generalization for spatial reinforcement learning

PLoS Computational Biology

Volumn 7, Issue 10, 2011, Pages

  Gustafson, Nicholas J ^a   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CELLS; CYTOLOGY; GEODESY; NEURAL NETWORKS; REINFORCEMENT LEARNING;

BRAIN MECHANISMS; COMPLEX TASK; EFFICIENT LEARNING; GENERALISATION; GRID CELLS; HIPPOCAMPAL PLACE CELLS; PLACE CELL; REINFORCEMENT LEARNINGS; SPATIAL NAVIGATION; THROUGH THE LENS;

LOCATION;

ANIMAL CELL; ARTICLE; BRAIN FUNCTION; CELL FUNCTION; CELL POPULATION; CELL TRACKING; CELLULAR DISTRIBUTION; CONTROLLED STUDY; HIPPOCAMPUS; NEUROANATOMY; NEURONAVIGATION; NONHUMAN; RAT; REINFORCEMENT; SIMULATION; TASK PERFORMANCE; ANIMAL; BEHAVIOR; BIOLOGICAL MODEL; COMPUTER SIMULATION; CYTOLOGY; ENTORHINAL CORTEX; MAZE TEST; NERVE CELL; PHYSIOLOGY;

ANIMALS; COMPUTER SIMULATION; ENTORHINAL CORTEX; HIPPOCAMPUS; MAZE LEARNING; MODELS, NEUROLOGICAL; NEURONS; RATS; REINFORCEMENT (PSYCHOLOGY); SPATIAL BEHAVIOR;

EID: 80055084825     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1002235     Document Type: Article

References (89)

1. O'Keefe J, Dostrovsky J, (1971) The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Res 34: 171-175.
2. Hafting T, Fyhn M, Molden S, Moser MB, Moser EI, (2005) Microstructure of a spatial map in the entorhinal cortex. Nature 436: 801-806.
3. Foster DJ, Morris RG, Dayan P, (2000) A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10: 1-16.
4. Redish AD, Touretzky DS, (1997) Cognitive maps beyond the hippocampus. Hippocampus 7: 15-35.
5. McNaughton BL, Barnes CA, Gerrard JL, Gothard K, Jung MW, et al. (1996) Deciphering the hippocampal polyglot: the hippocampus as a path integration system. J Exp Biol 199: 173-185.
6. Samsonovich A, McNaughton BL, (1997) Path integration and cognitive mapping in a continuous attractor neural network model. J Neurosci 17: 5900-5920.
7. Redish AD, Touretzky DS, (1998) The role of the hippocampus in solving the Morris water maze. Neural Comput 10: 73-111.
8. Sutton R, Barto A, (1998) Reinforcement learning: An introduction Cambridge, MA MIT Press.
9. Houk J, Adams J, Barto A, (1995) A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk J, Davis J, editors. Models of information processing in the basal ganglia Cambridge, MA MIT Press pp. 249-270.
10. Schultz W, Dayan P, Montague PR, (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
11. Doya K, (2008) Modulators of decision making. Nat Neurosci 11: 410-416.
12. Blair HT, Welday AC, Zhang K, (2007) Scale-invariant memory representations emerge from moire interference between grid fields that produce theta oscillations: a computational model. J Neurosci 27: 3211-3229.
13. Solstad T, Moser EI, Einevoll GT, (2006) From grid cells to place cells: a mathematical model. Hippocampus 16: 1026-1031.
14. Mahadevan S, (2005) Proto-value functions: Developmental reinforcement learning. Proceedings of the 22nd International Conference on Machine Learning; 7-11 August 2005; Bonn, Germany pp. 553-560 ICML 2005.
15. Mahadevan S, Maggioni M, (2006) Value Function Approximation with Diffusion Wavelets and Laplacian Eigenfunctions. Adv Neural Inf Process Syst 18: 843-850.
16. Dayan P, (1993) Improving Generalization for Temporal Difference Learning: The Successor Representation. Neural Comput 5: 613-624.
17. Konidaris G, Osentoski S, (2008) Value function approximation in reinforcement learning using the Fourier basis. Technical Report UM-CS-2008-19, Department of Computer Science, University of Massachusetts, Amherst.
18. Tenenbaum JB, de Silva V, Langford JC, (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290: 2319-2323.
19. Dragoi G, Harris KD, Buzsaki G, (2003) Place representation within hippocampal networks is modified by long-term potentiation. Neuron 39: 843-853.
20. McNaughton BL, Barnes CA, O'Keefe J, (1983) The contributions of position, direction, and velocity to single unit activity in the hippocampus of freely-moving rats. Exp Brain Res 52: 41-49.
21. Mehta MR, Quirk MC, Wilson MA, (2000) Experience-dependent asymmetric shape of hippocampal receptive fields. Neuron 25: 707-715.
22. O'Keefe J, Recce ML, (1993) Phase relationship between hippocampal place units and the EEG theta rhythm. Hippocampus 3: 317-330.
23. Muller RU, Bostock E, Taube JS, Kubie JL, (1994) On the directional firing properties of hippocampal place cells. J Neurosci 14: 7235-7251.
24. Derdikman D, Whitlock JR, Tsao A, Fyhn M, Hafting T, et al. (2009) Fragmentation of grid cell maps in a multicompartment environment. Nat Neurosci 12: 1325-1332.
25. Best PJ, White AM, Minai A, (2001) Spatial processing in the brain: the activity of hippocampal place cells. Annu Rev Neurosci 24: 459-486.
26. Muller RU, Stead M, Pach J, (1996) The hippocampus as a cognitive graph. J Gen Physiol 107: 663-694.
27. Blum KI, Abbott LF, (1996) A model of spatial map formation in the hippocampus of the rat. Neural Comput 8: 85-93.
28. Foster DJ, Wilson MA, (2006) Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440: 680-683.
29. Gerstner W, Abbott LF, (1997) Learning navigational maps through potentiation and modulation of hippocampal place cells. J Comput Neurosci 4: 79-94.
30. Kjelstrup KB, Solstad T, Brun VH, Hafting T, Leutgeb S, et al. (2008) Finite scale of spatial representation in the hippocampus. Science 321: 140-143.
31. Jung MW, Wiener SI, McNaughton BL, (1994) Comparison of spatial firing characteristics of units in dorsal and ventral hippocampus of the rat. J Neurosci 14: 7347-7356.
32. O'Keefe J, Burgess N, (1996) Geometric determinants of the place fields of hippocampal neurons. Nature 381: 425-428.
33. Brun VH, Solstad T, Kjelstrup KB, Fyhn M, Witter MP, et al. (2008) Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex. Hippocampus 18: 1200-1212.
34. Barry C, Hayman R, Burgess N, Jeffery KJ, (2007) Experience-dependent rescaling of entorhinal grids. Nat Neurosci 10: 682-684.
35. Fuhs MC, Touretzky DS, (2006) A spin glass model of path integration in rat medial entorhinal cortex. J Neurosci 26: 4266-4276.
36. Burgess N, Barry C, O'Keefe J, (2007) An oscillatory interference model of grid cell firing. Hippocampus 17: 801-812.
37. Fiete IR, Burak Y, Brookings T, (2008) What grid cells convey about rat location. J Neurosci 28: 6858-6871.
38. Hasselmo ME, (2008) Grid cell mechanisms and function: contributions of entorhinal persistent spiking and phase resetting. Hippocampus 18: 1213-1229.
39. Hasselmo ME, Giocomo LM, Zilli EA, (2007) Grid cell firing may arise from interference of theta frequency membrane potential oscillations in single neurons. Hippocampus 17: 1252-1271.
40. Hasselmo ME, Brandon MP, (2008) Linking cellular mechanisms to behavior: entorhinal persistent spiking and membrane potential oscillations may underlie path integration, grid cell firing, and episodic memory. Neural Plast 2008: 658323.
41. Mhatre H, Gorchetchnikov A, Grossberg S, (2010) Grid cell hexagonal patterns formed by fast self-organized learning within entorhinal cortex. Hippocampus.E-pub ahead of print. doi:10.1002/hipo.20901.
42. Zilli EA, Hasselmo ME, (2010) Coupled noisy spiking neurons as velocity-controlled oscillators in a model of grid cell spatial firing. J Neurosci 30: 13850-13860.
43. Rolls ET, Stringer SM, Elliot T, (2006) Entorhinal cortex grid cells can map to hippocampal place cells by competitive learning. Network 17: 447-465.
44. Franzius M, Vollgraf R, Wiskott L, (2007) From grids to places. J Comput Neurosci 22: 297-299.
45. Jeffery KJ, (2007) Self-localization and the entorhinal-hippocampal system. Curr Opin Neurobiol 17: 684-691.
46. Molter C, Yamaguchi Y, (2008) Impact of temporal coding of presynaptic entorhinal cortex grid cells on the formation of hippocampal place fields. Neural Netw 21: 303-310.
47. O'Keefe J, Burgess N, (2005) Dual phase and rate coding in hippocampal place cells: theoretical significance and relationship to entorhinal grid cells. Hippocampus 15: 853-866.
48. Moser EI, Kropff E, Moser MB, (2008) Place cells, grid cells, and the brain's spatial representation system. Annu Rev Neurosci 31: 69-89.
49. Suri RE, Schultz W, (2001) Temporal difference model reproduces anticipatory neural activity. Neural Comput 13: 841-862.
50. Brown MA, Sharp PE, (1995) Simulation of spatial learning in the Morris water maze by a neural network model of the hippocampal formation and nucleus accumbens. Hippocampus 5: 171-188.
51. Johnson A, Redish AD, (2005) Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw 18: 1163-1171.
52. Fyhn M, Molden S, Witter MP, Moser EI, Moser MB, (2004) Spatial representation in the entorhinal cortex. Science 305: 1258-1264.
53. Canto CB, Wouterlood FG, Witter MP, (2008) What does the anatomical organization of the entorhinal cortex tell us? Neural Plast 2008: 381243.
54. Sargolini F, Fyhn M, Hafting T, McNaughton BL, Witter MP, et al. (2006) Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science 312: 758-762.
55. Finch DM, Gigg J, Tan AM, Kosoyan OP, (1995) Neurophysiology and neuropharmacology of projections from entorhinal cortex to striatum in the rat. Brain Res 670: 233-247.
56. Krayniak PF, Meibach RC, Siegel A, (1981) A projection from the entorhinal cortex to the nucleus accumbens in the rat. Brain Res 209: 427-431.
57. Totterdell S, Meredith GE, (1997) Topographical organization of projections from the entorhinal cortex to the striatum of the rat. Neuroscience 78: 715-729.
58. Corbit LH, Ostlund SB, Balleine BW, (2002) Sensitivity to instrumental contingency degradation is mediated by the entorhinal cortex and its efferents via the dorsal hippocampus. J Neurosci 22: 10976-10984.
59. Langston RF, Ainge JA, Couey JJ, Canto CB, Bjerknes TL, et al. (2010) Development of the spatial representation system in the rat. Science 328: 1576-1580.
60. Wills TJ, Cacucci F, Burgess N, O'Keefe J, (2010) Development of the hippocampal cognitive map in preweanling rats. Science 328: 1573-1576.
61. Ludvig EA, Sutton RS, Kehoe EJ, (2008) Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput 20: 3034-3054.
62. Barry C, Burgess N, (2007) Learning in a geometric model of place cell firing. Hippocampus 17: 786-800.
63. Hasselmo ME, (2007) Arc length coding by interference of theta frequency oscillations may underlie context-dependent hippocampal unit data and episodic memory function. Learn Mem 14: 782-794.
64. Skaggs WE, McNaughton BL, (1998) Spatial firing properties of hippocampal CA1 populations in an environment containing two visually identical regions. J Neurosci 18: 8455-8466.
65. Muller RU, Kubie JL, (1987) The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. J Neurosci 7: 1951-1968.
66. Alvernhe A, Save E, Poucet B, (2011) Local remapping of place cell firing in the Tolman detour task. Eur J Neurosci 33: 1696-1705.
67. Tolman E, (1932) Purposive Behaviors in Animals and Men New York Appleton-Century-Crofts.
68. Collett TS, Cartwright BA, Smith BA, (1986) Landmark learning and visuo-spatial memories in gerbils. J Comp Physiol A 158: 835-851.
69. Solstad T, Boccara CN, Kropff E, Moser MB, Moser EI, (2008) Representation of geometric borders in the entorhinal cortex. Science 322: 1865-1868.
70. Sharp PE, (1991) Computer-Simulation of Hippocampal Place Cells. Psychobiology 19: 103-115.
71. Sharp PE, Blair HT, Brown M, (1996) Neural network modeling of the hippocampal formation spatial signals and their possible role in navigation: a modular approach. Hippocampus 6: 720-734.
72. Redish AD, (1999) Beyond the cognitive map: From place cells to episodic memory Cambridge, MA The MIT Press.
73. Gorchetchnikov A, Grossberg S, (2007) Space, time and learning in the hippocampus: how fine spatial and temporal scales are expanded into population codes for behavioral control. Neural Netw 20: 182-193.
74. Hasselmo ME, (2009) A model of episodic memory: mental time travel along encoded trajectories using grid cells. Neurobiol Learn Mem 92: 559-573.
75. McNaughton BL, Battaglia FP, Jensen O, Moser EI, Moser MB, (2006) Path integration and the neural basis of the 'cognitive map'. Nat Rev Neurosci 7: 663-678.
76. Jeanblanc J, Peterschmitt Y, Hoeltzel A, Louilot A, (2004) Influence of the entorhinal cortex on accumbal and striatal dopaminergic responses in a latent inhibition paradigm. Neuroscience 128: 187-200.
77. Sutton RS, (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning; 21-23 June 1990; Austin, Texas pp. 216-224 ICML 1990.
78. Daw N, Niv Y, Dayan P, (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
79. Johnson A, van der Meer MA, Redish AD, (2007) Integrating hippocampus and striatum in decision-making. Curr Opin Neurobiol 17: 692-697.
80. Hasselmo ME, (2005) A model of prefrontal cortical mechanisms for goal-directed behavior. J Cogn Neurosci 17: 1115-1129.
81. Johnson A, Redish AD, (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27: 12176-12189.
82. Doya K, (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw 12: 961-974.
83. Daw ND, Courville AC, Touretzky DS, (2006) Representation and timing in theories of the dopamine system. Neural Comput 18: 1637-1677.
84. Dayan P, Daw ND, (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8: 429-453.
85. Sutton R, (1988) Learning to predict by the methods of temporal differences. Mach Learn 3: 9-44.
86. McClure S, Daw N, Montague P, (2003) A computational substrate for incentive salience. Trends Neurosci 26: 423-428.
87. Fenton AA, Kao HY, Neymotin SA, Olypher A, Vayntrub Y, et al. (2008) Unmasking the CA1 ensemble place code by exposures to small and large environments: more place cells and multiple, irregularly arranged, and expanded place fields in the larger space. J Neurosci 28: 11250-11262.
88. Floyd R, (1962) Algorithm 97: Shortest path. Commun ACM 5: 345.
89. Sammon JW, (1969) A Nonlinear Mapping for Data Structure Analysis. IEEE Trans Comput 18: 401-409.