Interplay of approximate planning strategies

Proceedings of the National Academy of Sciences of the United States of America

Volumn 112, Issue 10, 2015, Pages 3098-3103

  Huys, Quentin J M ^a,b   Lally, Níall ^c,d   Faulkner, Paul ^e   Eshel, Neir ^f   Seifritz, Erich ^b   Gershman, Samuel J ^g   Dayan, Peter ^h   Roiser, Jonathan P ^c  

^a UNIVERSITY OF ZURICH   (Switzerland)

^b UNIVERSITY HOSPITAL OF PSYCHIATRY   (Switzerland)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

^d NATIONAL INSTITUTE OF MENTAL HEALTH   (United States)

^e UNIVERSITY OF CALIFORNIA   (United States)

^f HARVARD MEDICAL SCHOOL   (United States)

^g MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^h UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Hierarchical reinforcement learning; memoization; Planning; pruning

Indexed keywords

ACCURACY; ARTICLE; ARTIFICIAL INTELLIGENCE; BEHAVIORAL SCIENCE; COGNITION; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; DECISION MAKING; DECISION TREE; FEMALE; HUMAN; HUMAN EXPERIMENT; INTELLIGENCE QUOTIENT; MALE; PLANNING; PRIORITY JOURNAL; PROBABILITY; PROCESS MODEL; REINFORCEMENT; TASK PERFORMANCE; THEORETICAL STUDY; INTELLIGENCE; ORGANIZATION AND MANAGEMENT; STATISTICS;

HUMANS; INTELLIGENCE; PLANNING TECHNIQUES; STOCHASTIC PROCESSES;

EID: 84924325916     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1414219112     Document Type: Article

References (37)

1. Sutton RS, Precup D, Singh S (1999) Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artif Intell 112:181-211.
2. Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition 113(3):262-280.
3. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12):1704-1711.
4. Huys QJM, et al. (2012) Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLOS Comput Biol 8(3):e1002410.
5. Dezfouli A, Balleine BW (2012) Habits, action sequences and reinforcement learning. Eur J Neurosci 35(7):1036-1051.
6. Dezfouli A, Balleine BW (2013) Actions, action sequences and habits: Evidence that goal-directed and habitual action control are hierarchically organized. PLOS Comput Biol 9(12):e1003364.
7. Dietterich TG (2000) Hierarchical reinforcement learning with the MAXQ value function decomposition. J Artif Intell Res 13:227-303.
8. Jog MS, Kubota Y, Connolly CI, Hillegaart V, Graybiel AM (1999) Building neural representations of habits. Science 286(5445):1745-1749.
9. Koechlin E, Ody C, Kouneiher F (2003) The architecture of cognitive control in the human prefrontal cortex. Science 302(5648):1181-1185.
10. Badre D (2008) Cognitive control, hierarchy, and the rostro-caudal organization of the frontal lobes. Trends Cogn Sci 12(5):193-200.
11. Ostlund SB, Winterbauer NE, Balleine BW (2009) Evidence of action sequence chunking in goal-directed instrumental conditioning and its dependence on the dorsomedial prefrontal cortex. J Neurosci 29(25):8280-8287.
12. Logan GD, Crump MJC (2010) Cognitive illusions of authorship reveal hierarchical error detection in skilled typists. Science 330(6004):683-686.
13. Michie D (1968) Memo functions and machine learning. Nature 218:19-22.
14. O'Donnell TJ, Goodman ND, Tenenbaum JB (2009) Fragment grammars: Exploring computation and reuse in language. Technical Report MIT-CSAIL-TR-2009-013 (Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA).
15. O'Donnell TJ (2015) Productivity and Reuse in Language: A Theory of Linguistic Computation and Storage (MIT Press, Cambridge, MA).
16. Wingate D, Diuk C, O'Donnell T, Tenenbaum J, Gershman S (2013) Compositional policy priors. Technical Report MIT-CSAIL-TR 2013-007 (Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA).
17. Kass R, Raftery A (1995) Bayes factors. J Am Stat Assoc 90(430):773-795.
18. Teh YW, Jordan MI, Beal MJ, Blei DM (2006) Hierarchical Dirichlet processes. J Am Stat Assoc 101(476):1566-1581.
19. Keramati M, Dezfouli A, Piray P (2011) Speed/accuracy trade-off between the habitual and the goal-directed processes. PLOS Comput Biol 7(5):e1002055.
20. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).
21. Bhandari A, Duncan J (2014) Goal neglect and knowledge chunking in the construction of novel behaviour. Cognition 130(1):11-30.
22. Hirtle SC, Jonides J (1985) Evidence of hierarchies in cognitive maps. Mem Cognit 13(3):208-217.
23. McGovern A, Barto AG (2001) Automatic discovery of subgoals in reinforcement learning using diverse density. Proceedings of the Eighteenth International Conference on Machine Learning (Morgan Kaufmann, San Francisco), pp 361-368.
24. Wiener JM, Mallot HA (2003) 'Fine-to-coarse' route planning and navigation in regionalized environments. Spat Cogn Comput 3:331-358.
25. Şimşek Ö, Wolfe AP, Barto AG (2005) Identifying useful subgoals in reinforcement learning by local graph partitioning. Proceedings of the 22nd International Conference on Machine Learning (Assoc for Computing Machinery, New York), pp 816-823.
26. Opsahl T, Agneessens F, Skvoretz J (2010) Node centrality in weighted networks: Generalizing degree and shortest paths. Soc Networks 32:245-251.
27. Schapiro AC, Rogers TT, Cordova NI, Turk-Browne NB, Botvinick MM (2013) Neural representations of events arise from temporal community structure. Nat Neurosci 16(4):486-492.
28. Huys QJM, Guitart-Masip M, Dolan RJ, Dayan P (2015) Decision-theoretic psychiatry. Clin Psychol Sci, in press.
29. Watkins CJCH (1989) Learning from delayed rewards. PhD thesis (Cambridge Univ, Cambridge, UK).
30. Lau B, Glimcher PW (2005) Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav 84(3):555-579.
31. Seo H, Barraclough DJ, Lee D (2007) Dynamic signals related to choices and outcomes in the dorsolateral prefrontal cortex. Cereb Cortex 17(Suppl 1):i110-i117.
32. Camerer C, Ho TH (1998) Experience-weighted attraction learning in coordination games: Probability rules, heterogeneity, and time-variation. J Math Psychol 42(2/3):305-326.
33. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69(6):1204-1215.
34. Rutledge RB, et al. (2009) Dopaminergic drugs modulate learning rates and perseveration in Parkinson's patients in a dynamic foraging task. J Neurosci 29(48):15104-15114.
35. Seymour B, Daw ND, Roiser JP, Dayan P, Dolan R (2012) Serotonin selectively modulates reward value in human decision-making. J Neurosci 32(17):5833-5842.
36. Sheehan DV, et al. (1998) The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 59(Suppl 20):22-33, quiz 34-57.
37. Wechsler D (2001) Wechsler Test of Adult Reading Manual (The Psychological Corp, San Antonio, TX).