Neural Mechanisms of Hierarchical Planning in a Virtual Subway Network

Neuron

Volumn 90, Issue 4, 2016, Pages 893-903

  Balaguer, Jan ^a,c   Spiers, Hugo ^b   Hassabis, Demis ^c   Summerfield, Christopher ^a,c  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

^c DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; BEHAVIORAL SCIENCE; BOLD SIGNAL; CONTROLLED STUDY; DORSOMEDIAL PREFRONTAL CORTEX; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; FUNCTIONAL NEUROIMAGING; HIPPOCAMPUS; HUMAN; HUMAN EXPERIMENT; MALE; NERVE CELL MEMBRANE STEADY POTENTIAL; NORMAL HUMAN; PREFRONTAL CORTEX; PREMOTOR CORTEX; PRIORITY JOURNAL; SIGNAL PROCESSING; TASK PERFORMANCE; VENTROMEDIAL PREFRONTAL CORTEX; VIRTUAL REALITY; VIRTUAL SUBWAY NETWORK; BEHAVIOR; BRAIN; BRAIN MAPPING; IMAGE PROCESSING; NERVE CELL NETWORK; NUCLEAR MAGNETIC RESONANCE IMAGING; PHYSIOLOGY; PROCEDURES; YOUNG ADULT;

ADULT; BEHAVIOR; BRAIN; BRAIN MAPPING; FEMALE; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; MAGNETIC RESONANCE IMAGING; MALE; NERVE NET; TASK PERFORMANCE AND ANALYSIS; YOUNG ADULT;

EID: 84968867568     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2016.03.037     Document Type: Article

References (50)

1. Badre D., Kayser A.S., D'Esposito M. Frontal cortex and the discovery of abstract action rules. Neuron 2010, 66:315-326.
2. Botvinick M., Nystrom L.E., Fissell K., Carter C.S., Cohen J.D. Conflict monitoring versus selection-for-action in anterior cingulate cortex. Nature 1999, 402:179-181.
3. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
4. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
5. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
6. Dickinson A., Balleine B. The role of learning in motivation. Stevens' Handbook of Experimental Psychology Vol 3: Learning, Motivation and Emotion 2002, 497-533. Wiley. C.R. Gallistel (Ed.).
7. Dolan R.J., Dayan P. Goals and habits in the brain. Neuron 2013, 80:312-325.
8. Genovese C.R., Lazar N.A., Nichols T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 2002, 15:870-878.
9. Gershman S.J., Horvitz E.J., Tenenbaum J.B. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science 2015, 349:273-278.
10. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
11. Hayden B.Y., Pearson J.M., Platt M.L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 2011, 14:933-939.
12. Holroyd C.B., Yeung N. Motivation of extended behaviors by anterior cingulate cortex. Trends Cogn. Sci. 2012, 16:122-128.
13. Howard L.R., Javadi A.H., Yu Y., Mill R.D., Morrison L.C., Knight R., Loftus M.M., Staskute L., Spiers H.J. The hippocampus and entorhinal cortex encode the path and Euclidean distances to goals during navigation. Curr. Biol. 2014, 24:1331-1340.
14. Huys Q.J., Eshel N., O'Nions E., Sheridan L., Dayan P., Roiser J.P. Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol. 2012, 8:e1002410.
15. Huys Q.J., Lally N., Faulkner P., Eshel N., Seifritz E., Gershman S.J., Dayan P., Roiser J.P. Interplay of approximate planning strategies. Proc. Natl. Acad. Sci. USA 2015, 112:3098-3103.
16. Koechlin E., Jubault T. Broca's area and the hierarchical organization of human behavior. Neuron 2006, 50:963-974.
17. Koechlin E., Summerfield C. An information theoretical approach to prefrontal executive function. Trends Cogn. Sci. 2007, 11:229-235.
18. Koechlin E., Ody C., Kouneiher F. The architecture of cognitive control in the human prefrontal cortex. Science 2003, 302:1181-1185.
19. Kolling N., Behrens T.E., Mars R.B., Rushworth M.F. Neural mechanisms of foraging. Science 2012, 336:95-98.
20. Miller G.A., Galanter E., Pribram K.A. Plans and the Structure of Behavior 1960, Holt, Rhinehart, & Winston.
21. Morris R.G., Miotto E.C., Feigenbaum J.D., Bullock P., Polkey C.E. The effect of goal-subgoal conflict on planning ability after frontal- and temporal-lobe lesions in humans. Neuropsychologia 1997, 35:1147-1157.
22. Newcombe N., Liben L.S. Barrier effects in the cognitive maps of children and adults. J. Exp. Child Psychol. 1982, 34:46-58.
23. Norman D.A., Shallice T. Attention to action: willed and automatic control of behaviour. Consciousness and Self-Regulation 1986, Plenum Press. G.E. Schwartz, D. Shapiro (Eds.).
24. Owen A.M., Downes J.J., Sahakian B.J., Polkey C.E., Robbins T.W. Planning and spatial working memory following frontal lobe lesions in man. Neuropsychologia 1990, 28:1021-1034.
25. Penny W.D., Friston K., Ashburner J., Kiebel S., Nichols T.E. Statistical Parametric Mapping: The Analysis of Functional Brain Images 2006, Academic Press.
26. Ponsen, M., Taylor, M.E., and Tuyls, K. (2010). Abstraction and generalization in reinforcement learning: a summary and framework. R. Goebel, J. Siekmann, and W. Wahlster, eds. Proceedings of the AAMAS 2009 Conference in Budapest, 5924, 1-32.
27. Ribas-Fernandes J.J., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., Botvinick M.M. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71:370-379.
28. Russell S., Norvig P. Artificial Intelligence: A Modern Approach 1995, Prentice Hall.
29. Schacter D.L., Addis D.R. The cognitive neuroscience of constructive memory: remembering the past and imagining the future. Philos. Trans. R. Soc. Lond. B Biol. Sci. 2007, 362:773-786.
30. Schacter D.L., Addis D.R., Hassabis D., Martin V.C., Spreng R.N., Szpunar K.K. The future of memory: remembering, imagining, and the brain. Neuron 2012, 76:677-694.
31. Schapiro A.C., Rogers T.T., Cordova N.I., Turk-Browne N.B., Botvinick M.M. Neural representations of events arise from temporal community structure. Nat. Neurosci. 2013, 16:486-492.
32. Schoenbaum G., Roesch M.R., Stalnaker T.A., Takahashi Y.K. A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat. Rev. Neurosci. 2009, 10:885-892.
33. Shallice T. Specific impairments of planning. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1982, 298:199-209.
34. Shallice T., Burgess P.W. Deficits in strategy application following frontal lobe damage in man. Brain 1991, 114:727-741.
35. Shenhav A., Botvinick M.M., Cohen J.D. The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 2013, 79:217-240.
36. Silver D., Huang A., Maddison C.J., Guez A., Sifre L., van den Driessche G., Schrittwieser J., Antonoglou I., Panneershelvam V., Lanctot M., et al. Mastering the game of Go with deep neural networks and tree search. Nature 2016, 529:484-489.
37. Solway A., Diuk C., Córdova N., Yee D., Barto A.G., Niv Y., Botvinick M.M. Optimal behavioral hierarchy. PLoS Comput. Biol. 2014, 10:e1003779.
38. Stevens A., Coupe P. Distortions in judged spatial relations. Cognit. Psychol. 1978, 10:422-437.
39. Summerfield C., Koechlin E. Decision-making and prefrontal executive function. The Cognitive Neurosciences 2009, 1019-1030. MIT Press. M.S. Gazzaniga (Ed.).
40. Sutton R., Barto A. Reinforcement Learning 1998, MIT Press.
41. Tolman E.C. Cognitive maps in rats and men. Psychol. Rev. 1948, 55:189-208.
42. Tsetsos K., Wyart V., Shorkey S.P., Summerfield C. Neural mechanisms of economic commitment in the human medial prefrontal cortex. eLife 2014, 3:3.
43. Unterrainer J.M., Owen A.M. Planning and problem solving: from neuropsychology to functional neuroimaging. J. Physiol. Paris 2006, 99:308-317.
44. van den Heuvel O.A., Groenewegen H.J., Barkhof F., Lazeron R.H., van Dyck R., Veltman D.J. Frontostriatal system in planning complexity: a parametric functional magnetic resonance version of Tower of London task. Neuroimage 2003, 18:367-374.
45. Viard A., Doeller C.F., Hartley T., Bird C.M., Burgess N. Anterior hippocampus and goal-directed spatial decision making. J. Neurosci. 2011, 31:4613-4621.
46. Wagner G., Koch K., Reichenbach J.R., Sauer H., Schlösser R.G. The special involvement of the rostrolateral prefrontal cortex in planning abilities: an event-related fMRI study with the Tower of London paradigm. Neuropsychologia 2006, 44:2337-2347.
47. Ward G., Allport A. Planning and problem solving using the five disc Tower of London task. Q. J. Exp. Psychol. A 1997, 50:49-78.
48. Wiener J.M., Mallot H.A. 'Fine-to-Coarse' route planning and navigation in regionalized environments. Spat. Cogn. Comput. 2003, 3:331-358.
49. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 15:786-791.
50. Zeki S., Marini L. Three cortical stages of colour processing in the human brain. Brain 1998, 121:1669-1685.