Vector-based navigation using grid-like representations in artificial agents

Nature

Volumn 557, Issue 7705, 2018, Pages 429-433

  Banino, Andrea ^a,b   Barry, Caswell ^b   Uria, Benigno ^a   Blundell, Charles ^a   Lillicrap, Timothy ^a   Mirowski, Piotr ^a   Pritzel, Alexander ^a   Chadwick, Martin J ^a   Degris, Thomas ^a   Modayil, Joseph ^a   Wayne, Greg ^a   Soyer, Hubert ^a   Viola, Fabio ^a   Zhang, Brian ^a   Goroshin, Ross ^a   Rabinowitz, Neil ^a   Pascanu, Razvan ^a   Beattie, Charlie ^a   Petersen, Stig ^a   Sadik, Amir ^a   Gaffney, Stephen ^a   King, Helen ^a   Kavukcuoglu, Koray ^a   Hassabis, Demis ^a,c   Hadsell, Raia ^a   Kumaran, Dharshan ^a,b   more..

^a DEEPMIND   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL NEURAL NETWORK; EMERGENCE; MAMMAL; NAVIGATION; PERFORMANCE ASSESSMENT; SPATIAL VARIATION; VECTOR;

ALGORITHM; ARTICLE; ARTIFICIAL NEURAL NETWORK; CELL ACTIVATION; ENTORHINAL CORTEX; GRID CELL; NEUROMODULATION; PRIORITY JOURNAL; REINFORCEMENT; TASK PERFORMANCE; VECTOR BASED NAVIGATION; HUMAN; HUMAN CELL; LETTER; MAMMAL; NONHUMAN; REINFORCEMENT LEARNING (MACHINE LEARNING); ANIMAL; BIOMIMETICS; CYTOLOGY; ENVIRONMENT; MACHINE LEARNING; PHYSIOLOGY; PROCEDURES; SPATIAL ORIENTATION;

MAMMALIA;

ANIMALS; BIOMIMETICS; ENTORHINAL CORTEX; ENVIRONMENT; GRID CELLS; HUMANS; MACHINE LEARNING; NEURAL NETWORKS (COMPUTER); SPATIAL NAVIGATION;

EID: 85046892207     PISSN: 00280836     EISSN: 14764687     Source Type: Journal    
DOI: 10.1038/s41586-018-0102-6     Document Type: Article

References (49)

1. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436-444 (2015).
2. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484-489 (2016).
3. Oh, J., Chockalingam, V., Singh, S. P. & Lee, H. Control of memory, active perception, and action in Minecraft. Proc. Intl Conf. Machine Learning 48 (2016).
4. Kulkarni, T. D., Saeedi, A., Gautam, S. & Gershman, S. J. Deep successor reinforcement learning. Preprint at https://arxiv.org/abs/1606.02396 (2016).
5. Mirowski, P. et al. Learning to navigate in complex environments. Intl Conf. Learning Representations (2017).
6. Hafting, T., Fyhn, M., Molden, S., Moser, M.-B. & Moser, E. I. Microstructure of a spatial map in the entorhinal cortex. Nature 436, 801-806 (2005).
7. Fiete, I. R., Burak, Y. & Brookings, T. What grid cells convey about rat location. J. Neurosci. 28, 6858-6871 (2008).
8. Mathis, A., Herz, A. V. & Stemmler, M. Optimal population codes for space: grid cells outperform place cells. Neural Comput. 24, 2280-2317 (2012).
9. McNaughton, B. L., Battaglia, F. P., Jensen, O., Moser, E. I. & Moser, M.-B. Path integration and the neural basis of the 'cognitive map'. Nat. Rev. Neurosci. 7, 663-678 (2006).
10. Erdem, U. M. & Hasselmo, M. A goal-directed spatial navigation model using forward trajectory planning based on grid cells. Eur. J. Neurosci. 35, 916-931 (2012).
11. Bush, D., Barry, C., Manson, D. & Burgess, N. Using grid cells for navigation. Neuron 87, 507-520 (2015).
12. Barry, C. & Burgess, N. Neural mechanisms of self-location. Curr. Biol. 24, R330-R339 (2014).
13. Mittelstaedt, M.-L. & Mittelstaedt, H. Homing by path integration in a mammal. Naturwissenschaften 67, 566-567 (1980).
14. Bassett, J. P. & Taube, J. S. Neural correlates for angular head velocity in the rat dorsal tegmental nucleus. J. Neurosci. 21, 5740-5751 (2001).
15. Kropff, E., Carmichael, J. E., Moser, M.-B. & Moser, E. I. Speed cells in the medial entorhinal cortex. Nature 523, 419-424 (2015).
16. Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929-1958 (2014).
17. Wills, T. J., Cacucci, F., Burgess, N. & O'Keefe, J. Development of the hippocampal cognitive map in preweanling rats. Science 328, 1573-1576 (2010).
18. Langston, R. F. et al. Development of the spatial representation system in the rat. Science 328, 1576-1580 (2010).
19. Zhang, S.-J. et al. Optogenetic dissection of entorhinal-hippocampal functional connectivity. Science 340, 1232627 (2013).
20. Sargolini, F. et al. Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science 312, 758-762 (2006).
21. Barry, C., Hayman, R., Burgess, N. & Jeffery, K. J. Experience-dependent rescaling of entorhinal grids. Nat. Neurosci. 10, 682-684 (2007).
22. Stensola, H. et al. The entorhinal grid map is discretized. Nature 492, 72-78 (2012).
23. Stemmler, M., Mathis, A. & Herz, A. V. Connecting multiple spatial scales to decode the population activity of grid cells. Sci. Adv. 1, e1500816 (2015).
24. Doeller, C. F., Barry, C. & Burgess, N. Evidence for grid cells in a human memory network. Nature 463, 657-661 (2010).
25. Kanitscheider, I. & Fiete, I. Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems. Preprint at https://arxiv.org/abs/1609.09059 (2016).
26. Milford, M. J. & Wyeth, G. F. Mapping a suburb with a single camera using a biologically inspired slam system. IEEE Trans. Robot. 24, 1038-1053 (2008).
27. Hardcastle, K., Ganguli, S. & Giocomo, L. M. Environmental boundaries as an error correction mechanism for grid cells. Neuron 86, 827-839 (2015).
28. Chen, G., King, J. A., Burgess, N. & O'Keefe, J. How vision and movement combine in the hippocampal place code. Proc. Natl Acad. Sci. USA 110, 378-383 (2013).
29. Sarel, A., Finkelstein, A., Las, L. & Ulanovsky, N. Vectorial representation of spatial goals in the hippocampus of bats. Science 355, 176-180 (2017).
30. Dissanayake, M. G., Newman, P., Clark, S., Durrant-Whyte, H. F. & Csorba, M. A solution to the simultaneous localization and map building (slam) problem. IEEE Trans. Robot. Autom. 17, 229-241 (2001).
31. Raudies, F. & Hasselmo, M. E. Modeling boundary vector cell firing given optic flow as a cue. PLOS Comput. Biol. 8, e1002553 (2012).
32. Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735-1780 (1997).
33. Bridle, J. S. in Touretzky, D. S. (ed.) Advances in Neural Information Processing Systems 2 211-217 (Morgan-Kaufmann, 1990).
34. Elman, J. L. & McClelland, J. L. Exploiting lawful variability in the speech wave. Invariance and Variability in Speech Processes 1, 360-380 (1986).
35. Tieleman, T. & Hinton, G. Lecture 6.5 - RmsProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning (2012).
36. MacKay, D. J. A practical bayesian framework for backpropagation networks. Neural Comput. 4, 448-472 (1992).
37. Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training recurrent neural networks. Proc. 30th ICML 28, 1310-1318 (2013).
38. Ackley, D. H., Hinton, G. E. & Sejnowski, T. J. A learning algorithm for Boltzmann machines. Cogn. Sci. 9, 147-169 (1985).
39. Beattie, C. et al. Deepmind lab. Preprint at https://arxiv.org/abs/1612.03801 (2016).
40. Doeller, C. F., Barry, C. & Burgess, N. Evidence for grid cells in a human memory network. Nature 463, 657-661 (2010).
41. Mnih, V. et al. Asynchronous methods for deep reinforcement learning. In Proc. 33nd Intl Conf. Machine Learning 1928-1937 (2016).
42. Touretzky, D. S. & Redish, A. D. Theory of rodent navigation based on interacting representations of space. Hippocampus 6, 247-270 (1996).
43. Foster, D. J., Morris, R. G. & Dayan, P. A model of hippocampally dependent navigation, using the temporal difference learning rule. Hippocampus 10, 1-16 (2000).
44. Graves, A. et al. Hybrid computing using a neural network with dynamic external memory. Nature 538, 471-476 (2016).
45. Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529-533 (2015).
46. Lin, L.-J. Reinforcement learning for robots using neural networks. Technical Report (Carnegie-Mellon Univ. School of Computer Science, 1993).
47. Knight, R. et al. Weighted cue integration in the rodent head direction system. Phil. Trans. R. Soc. Lond. B 369, 20120512 (2013).
48. Solstad, T., Boccara, C. N., Kropff, E., Moser, M.-B. & Moser, E. I. Representation of geometric borders in the entorhinal cortex. Science 322, 1865-1868 (2008).
49. Barry, C. & Burgess, N. To be a grid cell: Shuffling procedures for determining gridness. Preprint at https://www.biorxiv.org/content/early/2017/12/08/230250 (2017).