The algorithmic anatomy of model-based evaluation

Philosophical Transactions of the Royal Society B: Biological Sciences

Volumn 369, Issue 1655, 2014, Pages

  Daw, Nathaniel D ^a   Dayan, Peter ^b  

^a NEW YORK UNIVERSITY   (United States)

^b UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Model based reasoning; Model free reasoning; Monte Carlo tree search; Orbitofrontal cortex; Reinforcement learning; Striatum

Indexed keywords

ALGORITHM; ANATOMY; ANIMAL; COMPUTER SIMULATION; LEARNING; MONTE CARLO ANALYSIS; NUMERICAL MODEL; PREDICTION; TWENTIETH CENTURY;

ANIMAL; BIOLOGICAL MODEL; BRAIN; HUMAN; LEARNING; MARKOV CHAIN; PHYSIOLOGY;

ANIMALS; BRAIN; HUMANS; LEARNING; MARKOV CHAINS; MODELS, NEUROLOGICAL;

EID: 84907545889     PISSN: 09628436     EISSN: 14712970     Source Type: Journal    
DOI: 10.1098/rstb.2013.0478     Document Type: Article

References (129)

1. Doya K. 1999 What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 12, 961–974. (doi:10.1016/S0893-6080(99)00046-5)
2. Sutton R., Barto AG. 1998 Reinforcement learning: an introduction (adaptive computation and machine learning). Cambridge, MA: MIT Press.
3. Adams C, Dickinson A. 1981 Actions and habits: variations in associative representations during instrumental learning. In Information processing in animals: memory mechanisms (eds N Spear, R Miller), pp. 143–165. Hillsdale, NJ: Lawrence Erlbaum Associates.
4. Daw ND, Niv Y, Dayan P. 2005 Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8, 1704–1711. (doi:10.1038/n1560)
5. Dickinson A, Balleine B. 2002 The role of learning in motivation. In Stevens’ handbook of experimental psychology, vol. 3 (ed. C Gallistel), pp. 497–533. New York, NY: Wiley.
6. Kahneman D. 2010 Thinking, fast and slow. New York, NY: Farrar, Straus and Giroux.
7. Stanovich K, West R. 2002 Individual differences in reasoning: implications for the rationality debate? In Heuristics and biases: the psychology of intuitive judgment (eds T Gilovich, D Griffin, D Kahneman), pp. 421–440. Cambridge, UK: Cambridge University Press.
8. Samuel A. 1959 Some studies in machine learning using the game of checkers. IBM J. Res. Dev. 3, 210–229. (doi:10.1147/rd.33.0210)
9. Barto A., Sutton R., Anderson CW. 1983 Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybernet. 834–846. (doi:10.1109/TSMC.1983.6313077)
10. Dezfouli A, Balleine BW. 2012 Habits, action sequences and reinforcement learning. Eur. J. Neurosci. 35, 1036–1051. (doi:10.1111/j. 1460-9568.2012.08050.x)
11. Dezfouli A, Lingawi NW, Balleine BW. 2014 Habits as action sequences: hierarchical action control and changes in outcome value. Phil. Trans. R. Soc. B 369, 20130482. (doi:10.1098/rstb.2013.0482)
12. Dolan RJ, Dayan P. 2013 Goals and habits in the brain. Neuron 80, 312–325. (doi:10.1016/j.neuron.2013.09.007)
13. Holland PC. 2004 Relations between Pavlovian – instrumental transfer and reinforcer devaluation. J. Exp. Psychol. Anim. Behav. Process. 30, 104–117. (doi:10.1037/0097-7403.30.2.104)
14. Simon DA, Daw ND. 2011 Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci. 31, 5526– 5539. (doi:10.1523/JNEUROSCI.4647-10.2011)
15. Thorndike E. 1911 Animal intelligence. New York, NY: Macmillan.
16. Tolman E. 1949 There is more than one kind of learning. Psychol. Rev. 56, 144–155. (doi:10.1037/h0055304)
17. Otto AR, Gershman SJ, Markman AB, Daw ND. 2013 The curse of planning: dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol. Sci. 24, 751–761. (doi:10.1177/0956797612463080)
18. Otto AR, Raio CM, Chiang A, Phelps EA, Daw ND. 2013 Working-memory capacity protects model-based learning from stress. Proc. Natl Acad. Sci. SA 110, 20 941– 20 946. (doi:10.1073/pnas.1312011110)
19. Barto A. 1995 Adaptive critics and the basal ganglia. In Models of information processing in the basal ganglia (eds J Houk, J Davis, D Beiser),pp. 215–232. Cambridge, MA: MIT Press.
20. Montague PR, Dayan P, Sejnowski TJ. 1996 A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947.
21. Schultz W, Dayan P, Montague PR. 1997 A neural substrate of prediction and reward. Science 275, 1593– 1599. (doi:10.1126/science. 275.5306.1593)
22. Balleine BW. 2005 Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav. 86, 717–730. (doi:10.1016/j.physbeh.2005.08.061)
23. Killcross S, Coutureau E. 2003 Coordination of actions and habits in the medial prefrontal cortex of rats. Cereb. Cortex 13, 400– 408. (doi:10.1093/cercor/13.4.400)
24. McDannald MA, Jones JL, Takahashi YK, Schoenbaum G. 2013 Learning theory: a driving force in understanding orbitofrontal function. Neurobiol. Learn. Mem. 108C, 22–27.
25. McDannald MA, Lucantonio F, Burke KA, Niv Y, Schoenbaum G. 2011 Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 31, 2700–2705. (doi:10.1523/ JNEUROSCI.5499-10.2011)
26. McDannald MA, Takahashi YK, Lopatina N, Pietras BW, Jones JL, Schoenbaum G. 2012 Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 35, 991–996. (doi:10.1111/j.1460-9568.2011.07982.x)
27. Yin HH, Ostlund SB, Balleine BW. 2008 Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur. J. Neurosci. 28, 1437–1448. (doi:10.1111/j.1460-9568.2008.06422.x)
28. Glascher J, Daw N, Dayan P, O’Doherty JP. 2010 States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595. (doi:10.1016/j.neuron.2010.04.016)
29. Tricomi E, Balleine B., O’Doherty JP. 2009 A specific role for posterior dorsolateral striatum in human habit learning. Eur. J. Neurosci. 29, 2225–2232. (doi:10.1111/j.1460-9568.2009.06796.x)
30. Valentin VV, Dickinson A, O’Doherty JP. 2007 Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 27, 4019– 4026. (doi:10.1523/JNEUROSCI.0564-07.2007)
31. Wunderlich K, Dayan P, Dolan RJ. 2012 Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 15, 786–791. (doi:10.1038/nn.3068)
32. Keramati M, Dezfouli A, Piray P. 2011 Speed/ accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput. Biol. 7, e1002055. (doi:10.1371/journal.pcbi.1002055)
33. Pezzulo G, Rigoli F, Chersi F. 2013 The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front. Psychol. 4, 92. (doi:10.3389/fpsyg.2013.00092)
34. Solway A, Botvinick MM. 2012 Goal-directed decision making as probabilistic inference: a computational framework and potential neural correlates. Psychol. Rev. 119, 120–154. (doi:10.1037/a0026435)
35. Baum EB. 2004 What is thought? Cambridge, MA: MIT Press.
36. Dayan P. 2012 How to set the switches on this thing. Curr. Opin. Neurobiol. 22, 1068–1074. (doi:10.1016/j.conb.2012.05.011)
37. Russell S., Wefald EH. 1991 Do the right thing: studies in limited rationality. Cambridge, MA: MIT Press.
38. Simon HA. 1982 Models of bounded rationality. Cambridge, MA: MIT Press.
39. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ. 2011 Model-based influences on humans’ choices and striatal prediction errors. Neuron 69, 1204–1215. (doi:10.1016/j.neuron.2011.02.027)
40. Doll BB, Jacobs WJ, Sanfey AG, Frank MJ. 2009 Instructional control of reinforcement learning: a behavioral and neurocomputational investigation. Brain Res. 1299, 74–94. (doi:10.1016/j.brainres.2009.07.007)
41. Doll BB, Simon DA, Daw ND. 2012 The ubiquity of model-based reinforcement learning. Curr. Opin. Neurobiol. 22, 1075–1081. (doi:10.1016/j.conb.2012.08.003)
42. Gershman SJ, Markman AB, Otto AR. 2012 Retrospective revaluation in sequential decision making: a tale of two systems. J. Exp. Psychol. Gen. 24, 751–761.
43. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O’Donnell P, Niv Y, Schoenbaum G. 2011 Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 14, 1590–1597. (doi:10.1038/nn.2957)
44. Kaelbling LP, Littman ML, Cassandra AR. 1998 Planning and acting in partially observable stochastic domains. Artif. Intell. 101, 99–134. (doi:10.1016/S0004-3702(98)00023-X)
45. Kearns M, Mansour Y, Ng AY. 2002 A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Mach. Learn. 49, 193–208. (doi:10.1023/A:1017932429737)
46. Bertsekas DP, Tsitsiklis JN. 1996 Neuro-dynamic programming. Belmont, MA: Athena Scientific.
47. Sutton R. 1988 Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9–44.
48. Kocsis L, Szepesvari C. 2006 Bandit based Monte Carlo planning. In Machine learning: ECML 2006 (eds J Fu¨rnkranz, T Scheffer, M Spiliopoulou), pp. 282–293. Berlin, Germany: Springer.
49. Silver D, Sutton R., Muller M. 2008 Sample-based learning and search with permanent and transient memories. In Proc. 25th Int. Conf. on Machine Learning, pp. 968–975. New York, NY: Association for Computing Machinery.
50. Sutton R. 1990 Integrated architectures for learning, planning, reacting based on approximating dynamic programming. Proc. Seventh Int. Conf. on Machine Learning, pp. 216–224. San Fransisco, CA: Morgan Kaufmann.
51. Baxter J, Tridgell A, Weaver L. 2000 Learning to play chess using temporal differences. Mach. Learn. 40, 243– 263. (doi:10.1023/A:1007634325138)
52. Veness J, Silver D, Uther WT, Blair A. 2009 Bootstrapping from game tree search. In NIPS, vol. 19 (eds Y Bengio, D Schuurmans, J Lafferty, C Williams, A Culotta), pp. 1937–1945. Red Hook, NY: Curran Associates.
53. Dayan P. 1993 Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5, 613–624. (doi:10.1162/neco.1993.5.4.613)
54. Sutton RS. 1995 TD models: modeling the world at a mixture of time scales. In ICML, vol. 12 (eds A Prieditis, SJ Russell), pp. 531–539. San Mateo, CA: Morgan Kaufmann.
55. Silver D, Ciosek K. 2012 Compositional planning using optimal option models. In Proc. 29th Int. Conf. on Machine Learning, ICML ‘12 (eds J Langford, J Pineau), pp. 1063–1070. New York, NY: Omni Press.
56. Botvinick M, Weinstein A. 2014 Model-based hierarchical reinforcement learning and human action control. Phil. Trans. R. Soc. B 369, 20130480. (doi:10.1098/rstb.2013.0480)
57. Sutton R., Precup D, Singh S. 1999 Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112, 181–211. (doi:10.1016/S0004-3702(99) 00052-1)
58. Ng AY, Parr R, Koller D. 1999 Policy search via density estimation. In NIPS (eds SA Solla, TK Leen, K Müller), pp. 1022–1028. Cambridge, MA: MIT Press.
59. Wang T, Bowling M, Schuurmans D. 2007 Dual representations for dynamic programming and reinforcement learning. In IEEE Int. Symp. on Approximate Dynamic Programming and Reinforcement Learning, Honolulu, HI, 1–5 April 2007. ADPRL 2007, pp. 44–51. IEEE. (doi:10.1109/ADPRL.2007.368168)
60. Dickinson A. 1985 Actions and habits: the development of behavioural autonomy. Phil. Trans R. Soc. Lond. B 308, 67–78. (doi:10.1098/rstb.1985.0010)
61. Dearden R, Friedman N, Russell S. 1998 Bayesian Q-learning. In Proc. Fifteenth Nat. Conf. on Artificial Intelligence, pp. 761–768. Menlo Park, CA: American Association for Artificial Intelligence.
62. Engel Y, Mannor S, Meir R. 2003 Bayes meets Bellman: the Gaussian process approach to temporal difference learning. In ICML (eds T Fawcett, N Mishra), pp. 154–161. Washington, DC: AAAI Press.
63. Mannor S, Simester D, Sun P, Tsitsiklis JN. 2004 Bias and variance in value function estimation. In Proc. 21st Int. Conf. on Machine Learning, p. 72. New York, NY: Association for Computing Machinery.
64. Moore AW, Atkeson CG. 1993 Prioritized sweeping: reinforcement learning with less data and less time. Mach. Learn. 13, 103–130. (doi:10.1007/BF00993104)
65. Adams CD. 1982 Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol. 34, 77–98.
66. Watkins CJCH. 1989 Learning from delayed rewards. PhD thesis, Cambridge University, Cambridge, UK.
67. Hay N, Russell SJ. 2011 Metareasoning for Monte Carlo tree search. Technical Report no. UCB/EECS-2011-119. EECS Department, University of California, Berkeley, CA.
68. Bellman RE. 1957 Dynamic programming. Princeton, NJ: Princeton University Press.
69. Balleine B., O’Doherty JP. 2010 Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35, 48–69. (doi:10.1038/npp.2009.131)
70. Devan BD, Hong NS, McDonald RJ. 2011 Parallel associative processing in the dorsal striatum: segregation of stimulus–response and cognitive control subregions. Neurobiol. Learn. Mem. 96, 95–120. (doi:10.1016/j.nlm.2011.06.002)
71. Thorn CA, Atallah H, Howe M, Graybiel AM. 2010 Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron 66, 781–795. (doi:10.1016/j.neuron.2010.04.036)
72. Alexander GE, Crutcher MD. 1990 Functional architecture of basal ganglia circuits: neural substrates of parallel processing. Trends Neurosci. 13, 266–271. (doi:10.1016/0166-2236(90)90107-L)
73. Alexander GE, DeLong MR, Strick PL. 1986 Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci. 9, 357–381. (doi:10.1146/annurev.ne.09.030186.002041)
74. Frank MJ, Claus ED. 2006 Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, reversal. Psychol. Rev. 113, 300–326. (doi:10.1037/0033-295X.113.2.300)
75. Balleine B., Dickinson A. 1998 Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37, 407–419. (doi:10.1016/S0028-3908(98)00033-1)
76. Corbit LH, Muir JL, Balleine BW. 2003 Lesions of mediodorsal thalamus and anterior thalamic nuclei produce dissociable effects on instrumental conditioning in rats. Eur. J. Neurosci. 18, 1286–1294. (doi:10.1046/j. 1460-9568.2003.02833.x)
77. O’Doherty JP. 2007 Lights, camembert, action! The role of human orbitofrontal cortex in encoding stimuli, rewards, choices. Ann. NY Acad. Sci. 1121, 254–272. (doi:10.1196/annals.1401.036)
78. de Wit S, Watson P, Harsay HA, Cohen MX, van de Vijver I, Ridderinkhof KR. 2012 Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. J. Neurosci. 32, 12 066–12 075. (doi:10.1523/JNEUROSCI.1088-12.2012)
79. Liljeholm M, Tricomi E, O’Doherty JP, Balleine BW. 2011 Neural correlates of instrumental contingency learning: differential effects of action–reward conjunction and disjunction. J. Neurosci. 31, 2474–2480. (doi:10.1523/JNEUROSCI.3354-10.2011)
80. Dayan P, Berridge K. 2014 Model-based and model-free Pavlovian reward learning: revaluation, revision and revelation. Cogn. Affect. Behav. Neurosci. 14, 473–492. (doi:10.3758/s13415-014-0277-8)
81. Prevost C, Liljeholm M, Tyszka JM, O’Doherty JP. 2012 Neural correlates of specific and general Pavlovian-to-instrumental transfer within human amygdalar subregions: a high-resolution fMRI study. J. Neurosci. 32, 8383–8390. (doi:10.1523/JNEUROSCI.6237-11.2012)
82. Colwill RM, Rescorla RA. 1988 Associations between the discriminative stimulus and the reinforcer in instrumental learning. J. Exp. Psychol. Anim. Behav. Process. 14, 155–164. (doi:10.1037/0097-7403.14.2.155)
83. Estes W. 1943 Discriminative conditioning. I. A discriminative property of conditioned anticipation. J. Exp. Psychol. 32, 150–155. (doi:10.1037/h0058316)
84. Hart G, Leung BK, Balleine BW. 2013 Dorsal and ventral streams: the distinct role of striatal subregions in the acquisition and performance of goal-directed actions. Neurobiol. Learn. Mem. 108, 104–118. (doi:10.1016/j.nlm.2013.11.003)
85. Rescorla RA, Solomon RL. 1967 Two-process learning theory: relationships between Pavlovian conditioning and instrumental learning. Psychol. Rev. 74, 151–182. (doi:10.1037/h0024475)
86. Corbit LH, Leung BK, Balleine BW. 2013 The role of the amygdala–striatal pathway in the acquisition and performance of goal-directed instrumental actions. J. Neurosci. 33, 17 682– 17 690. (doi:10. 1523/JNEUROSCI.3271-13.2013)
87. Lingawi NW, Balleine BW. 2012 Amygdala central nucleus interacts with dorsolateral striatum to regulate the acquisition of habits. J. Neurosci. 323, 1073–1081. (doi:10.1523/JNEUROSCI.4806-11.2012)
88. Wiecki TV, Frank MJ. 2010 Neurocomputational models of motor and cognitive deficits in Parkinson’s disease. Prog. Brain Res. 183, 275–297. (doi:10.1016/S0079-6123(10)83014-6)
89. Foster DJ, Wilson MA. 2006 Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440, 680–683. (doi:10.1038/nature04587)
90. Foster DJ, Wilson MA. 2007 Hippocampal theta sequences. Hippocampus 17, 1093–1099. (doi:10.1002/hipo.20345)
91. Johnson A, van der Meer MAA, Redish AD. 2007 Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol. 17, 692–697. (doi:10.1016/j.conb.2008.01.003)
92. Pfeiffer BE, Foster DJ. 2013 Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497, 74–79. (doi:10.1038/nature12112)
93. Jones MW, Wilson MA. 2005 Theta rhythms coordinate hippocampal–prefrontal interactions in a spatial memory task. PLoS Biol. 3, e402. (doi:10.1371/journal.pbio.0030402)
94. Lansink CS, Goltstein PM, Lankelma JV, McNaughton BL, Pennartz CM. 2009 Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biol. 7, e1000173. (doi:10.1371/journal.pbio.1000173)
95. Doya K, Ishii S, Pouget A, Rao RP. (eds) 2007 Bayesian brain: probabilistic approaches to neural coding. Cambridge, MA: MIT Press.
96. Rainer G, Rao SC, Miller EK. 1999 Prospective coding for objects in primate prefrontal cortex. J. Neurosci. 19, 5493–5505.
97. Sakai K, Miyashita Y. 1991 Neural organization for the long-term memory of paired associates. Nature 354,152–155. (doi:10.1038/354152a0)
98. Faure A, Haberland U, Conde´ F, El Massioui N. 2005 Lesion to the nigrostriatal dopamine system disrupts stimulus–response habit formation. J. Neurosci. 25, 2771–2780. (doi:10.1523/JNEUROSCI.3894-04.2005)
99. Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH. 2013 A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966–973. (doi:10.1038/nn.3413)
100. Wang LP, Li F, Wang D, Xie K, Wang D, Shen X, Tsien JZ. 2011 NMDA receptors in dopaminergic neurons are crucial for habit learning. Neuron 72, 1055–1066. (doi:10.1016/j.neuron.2011.10.019)
101. Dickinson A, Smith J, Mirenowicz J. 2000 Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav. Neurosci. 114, 468–483. (doi:10.1037/0735-7044.114.3.468)
102. Robinson MJF, Berridge KC. 2013 Instant transformation of learned repulsion into motivational ‘wanting’. Curr. Biol. 23, 282–289. (doi:10.1016/j.cub.2013.01.016)
103. Botvinick M., Cohen J. 2014 The computational and neural basis of cognitive control: charted territory and new frontiers. Cogn. Sci. (doi:10.1111/cogs.12126)
104. Frank MJ, Loughry B, O’Reilly RC. 2001 Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn. Affect. Behav. Neurosci. 1, 137–160. (doi:10.3758/CABN.1.2.137)
105. Hazy TE, Frank MJ, O’Reilly RC. 2006 Banishing the homunculus: making working memory work. Neuroscience 139, 105–118. (doi:10.1016/j.neuroscience.2005.04.067)
106. Miller EK, Cohen JD. 2001 An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci. 24, 167–202. (doi:10.1146/annurev.neuro.24.1.167)
107. O’Reilly R, Braver T, Cohen J. 1999 A biologically based computational model of working memory. In Models of working memory: mechanisms of active maintenance and executive control (eds A Mikaye, P Shah), pp. 375–411. New York, NY: Cambridge University Press.
108. O’Reilly RC, Frank MJ. 2006 Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput. 18, 283–328. (doi:10.1162/0899766067 75093909)
109. O’Reilly RC, Noelle DC, Braver TS, Cohen J.. 2002 Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. Cereb. Cortex 12, 246–257. (doi:10.1093/ cercor/12.3.246)
110. Rougier NP, Noelle DC, Braver TS, Cohen J., O’Reilly RC. 2005 Prefrontal cortex and flexible cognitive control: rules without symbols. Proc. Natl Acad. Sci. USA 102, 7338–7343. (doi:10.1073/pnas.0502455102)
111. Howes A, Lewis RL, Vera A. 2009 Rational adaptation under task and processing constraints: implications for testing theories of cognition and action. Psychol. Rev. 116, 717–751. (doi:10.1037/a0017187)
112. Lewis RL, Howes A, Singh S. 2014 Computational rationality: linking mechanism and behavior through utility maximization. Top. Cogn. Sci. 6, 279–311. (doi:10.1111/tops.12086)
113. Raby CR, Alexis DM, Dickinson A, Clayton NS. 2007 Planning for the future by western scrub-jays. Nature 445, 919–921. (doi:10.1038/nature05575)
114. Loewenstein G, Prelec D. 1992 Anomalies in intertemporal choice: evidence and an interpretation. Q. J. Econ. 107, 573–597. (doi:10. 2307/2118482)
115. Guthrie E. 1935 The psychology of learning. New York, NY: Harper.
116. Botvinick M., Niv Y, Barto AC. 2009 Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113, 262–280. (doi:10.1016/j.cognition.2008.08.011)
117. Ribas-Fernandes JJF, Solway A, Diuk C, McGuire JT, Barto A., Niv Y, Botvinick MM. 2011 A neural signature of hierarchical reinforcement learning. Neuron 71, 370–379. (doi:10.1016/j.neuron.2011.05. 042)
118. Fermin A, Yoshida T, Ito M, Yoshimoto J, Doya K. 2010 Evidence for model-based action planning in a sequential finger movement task. J. Motiv. Behav. 42, 371–379. (doi:10.1080/00222895.2010.526467)
119. Kool W, Botvinick M. 2013 The intrinsic cost of cognitive control. Behav. Brain Sci. 36, 697–698. (doi:10.1017/S0140525X1300109X)
120. Shenhav A, Botvinick M., Cohen JD. 2013 The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron 79, 217–240. (doi:10.1016/j.neuron.2013.07.007)
121. Michie D, Chambers R. 1968 Boxes: an experiment in adaptive control. Mach. Intell. 2, 137–152.
122. Minsky M. 1952 A neural-analogue calculator based upon a probability model of reinforcement. Cambridge, MA: Harvard University, Psychological Laboratories.
123. Charness N. 1992 The impact of chess research on cognitive science. Psychol. Res. 54, 4–9. (doi:10. 1007/BF01359217)
124. de Groot AD. 1965 Thought and choice in chess. The Hague, The Netherlands: Mouton.
125. Newell A, Shaw JC, Simon HA. 1958 Chess-playing programs and the problem of complexity. IBM J. Res. Dev. 2, 320–335. (doi:10.1147/rd.24.0320)
126. Newell A et al. 1972 Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
127. Anderson JR. 1993 Problem solving and learning. Am. Psychol. 48, 35–44. (doi:10.1037/0003-066X.48.1.35)
128. Simon HA. 1975 The functional equivalence of problem solving skills. Cogn. Psychol. 7, 268–288. (doi:10.1016/0010-0285 (75)90012-2)
129. Huys QJ, Eshel N, O’Nions E, Sheridan L, Dayan P, Roiser JP. 2012 Bonsai trees in your head: how the Pavlovian system sculpts goal- directed choices by pruning decision trees. PLoS Comput. Biol. 8, e1002410. (doi:10.1371/journal.pcbi.1002410)