Evidence integration in model-based tree search

Proceedings of the National Academy of Sciences of the United States of America

Volumn 112, Issue 37, 2015, Pages 11708-11713

  Solway, Alec ^a   Botvinick, Matthew M ^a,b  

^a PRINCETON UNIVERSITY   (United States)

^b DEEPMIND   (United Kingdom)

Author keywords

Drift diffusion model; Reinforcement learning; Reward based decision making

Indexed keywords

ARTICLE; ARTIFICIAL INTELLIGENCE; CONTROLLED STUDY; DECISION MAKING; DECISION TREE; EVIDENCE INTEGRATION MODEL; HUMAN; PRIORITY JOURNAL; PROCESS MODEL; RESPONSE TIME; REWARD; SLOPE FACTOR; BAYES THEOREM; BIOLOGICAL MODEL; COMPUTER SIMULATION; LEARNING; REINFORCEMENT; REPRODUCIBILITY;

BAYES THEOREM; CHOICE BEHAVIOR; COMPUTER SIMULATION; DECISION MAKING; HUMANS; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); REPRODUCIBILITY OF RESULTS; REWARD;

EID: 84941703217     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1505483112     Document Type: Article

References (57)

1. Busemeyer JR, Townsend JT (1993) Decision field theory: A dynamic-cognitive approach to decision making in an uncertain environment. Psychol Rev 100(3):432-459.
2. Milosavljevic M, Malmaud J, Huth A, Koch C, Rangel A (2010) The drift diffusion model can account for the accuracy and reaction time of value-based choices under high and low time pressure. Judgm Decis Mak 5(6):437-449.
3. Krajbich I, Armel C, Rangel A (2010) Visual fixations and the computation and comparison of value in simple choice. Nat Neurosci 13(10):1292-1298.
4. Krajbich I, Rangel A (2011) Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions. Proc Natl Acad Sci USA 108(33):13852-13857.
5. Woodford M (2014) Stochastic choice: An optimizing neuroeconomic model. Am Econ Rev 104(5):495-500.
6. Daw ND (2012) Model-based reinforcement learning as cognitive search: neuro-computational theories. Cognitive Search: Evolution Algorithms and the Brain, eds Todd PM, Hills TT, Robbins TW (MIT Press, Cambridge, MA), pp 195-208.
7. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8(12): 1704-1711.
8. Dolan RJ, Dayan P (2013) Goals and habits in the brain. Neuron 80(2):312-325.
9. Huys QJM, et al. (2012) Bonsai trees in your head: How the pavlovian system sculpts goal-directed choices by pruning decision trees. PLOS Comput Biol 8(3):e1002410.
10. Huys QJM, et al. (2015) Interplay of approximate planning strategies. Proc Natl Acad Sci USA 112(10):3098-3103.
11. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA).
12. Bornstein AM, Daw ND (2012) Dissociating hippocampal and striatal contributions to sequential prediction learning. Eur J Neurosci 35(7):1011-1023.
13. Bornstein AM, Daw ND (2013) Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans. PLOS Comput Biol 9(12): e1003387.
14. Gläscher J, Daw N, Dayan P, O'Doherty JP (2010) States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585-595.
15. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69(6):1204-1215.
16. Simon DA, Daw ND (2011) Neural correlates of forward planning in a spatial decision task in humans. J Neurosci 31(14):5526-5539.
17. Wunderlich K, Dayan P, Dolan RJ (2012) Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 15(5):786-791.
18. Otto AR, Gershman SJ, Markman AB, Daw ND (2013) The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol Sci 24(5):751-761.
19. Otto AR, Raio CM, Chiang A, Phelps EA, Daw ND (2013) Working-memory capacity protects model-based learning from stress. Proc Natl Acad Sci USA 110(52):20941-20946.
20. Smittenaar P, FitzGerald THB, Romei V, Wright ND, Dolan RJ (2013) Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron 80(4):914-919.
21. Smittenaar P, Prichard G, FitzGerald THB, Diedrichsen J, Dolan RJ (2014) Transcranial direct current stimulation of right dorsolateral prefrontal cortex does not affect model-based or model-free reinforcement learning in humans. PLoS One 9(1):e86850.
22. Wunderlich K, Smittenaar P, Dolan RJ (2012) Dopamine enhances model-based over model-free choice behavior. Neuron 75(3):418-424.
23. Solway A, Botvinick MM (2012) Goal-directed decision making as probabilistic inference: A computational framework and potential neural correlates. Psychol Rev 119(1):120-154.
24. Diederich A (1997) Dynamic stochastic models for decision making under time constraints. J Math Psychol 41(3):260-274.
25. Roe RM, Busemeyer JR, Townsend JT (2001) Multialternative decision field theory: A dynamic connectionist model of decision making. Psychol Rev 108(2):370-392.
26. Usher M, McClelland JL (2001) The time course of perceptual choice: The leaky, competing accumulator model. Psychol Rev 108(3):550-592.
27. Teodorescu AR, Usher M (2013) Disentangling decision models: From independence to competition. Psychol Rev 120(1):1-38.
28. Morey RD (2008) Confidence intervals from normalized data: A correction to Cous-ineau (2005). Tutor Quant Methods Psychol 4(2):61-64.
29. Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461-464.
30. Boutilier C, Dean T, Hanks S (1999) Decision-theoretic planning: Structural assumptions and computational leverage. J Artif Intell Res 11(1):1-94.
31. Niv Y, Daw N, Dayan P (2005) How fast to work: Response vigor, motivation and tonic dopamine. Advances in Neural Information Processing Systems, eds Weiss Y, Schölkopf B, Platt J (MIT Press, Cambridge, MA), Vol 18, pp 1019-1026.
32. Niv Y, Daw ND, Joel D, Dayan P (2007) Tonic dopamine: Opportunity costs and the control of response vigor. Psychopharmacology (Berl) 191(3):507-520.
33. Kawagoe R, Takikawa Y, Hikosaka O (2004) Reward-predicting activity of dopamine and caudate neurons-a possible mechanism of motivational control of saccadic eye movement. J Neurophysiol 91(2):1013-1024.
34. Takikawa Y, Kawagoe R, Itoh H, Nakahara H, Hikosaka O (2002) Modulation of sac-cadic eye movements by predicted reward outcome. Exp Brain Res 142(2):284-291.
35. Resulaj A, Kiani R, Wolpert DM, Shadlen MN (2009) Changes of mind in decision-making. Nature 461(7261):263-266.
36. Ratcliff R (1978) A theory of memory retrieval. Psychol Rev 85(2):59-108.
37. Polyn SM, Norman KA, Kahana MJ (2009) A context maintenance and retrieval model of organizational processes in free recall. Psychol Rev 116(1):129-156.
38. Sederberg PB, Howard MW, Kahana MJ (2008) A context-based theory of recency and contiguity in free recall. Psychol Rev 115(4):893-912.
39. Ratcliff R, McKoon G (2008) The diffusion decision model: Theory and data for two-choice decision tasks. Neural Comput 20(4):873-922.
40. Wagenmakers E-J, Ratcliff R, Gomez P, McKoon G (2008) A diffusion model account of criterion shifts in the lexical decision task. J Mem Lang 58(1):140-159.
41. Wong KF, Wang XJ (2006) A recurrent network mechanism of time integration in perceptual decisions. J Neurosci 26(4):1314-1328.
42. Coulom R (2007) Efficient selectivity and backup operators in Monte-carlo tree search. Computers and Games, eds van den Herik HJ, Ciancarini P, Donkers HHLM (Springer, New York), pp 72-83.
43. Gelly S, Silver D (2011) Monte-carlo tree search and rapid action value estimation in computer Go. Artif Intell 175(11):1856-1875.
44. Kocsis L, Szepesvári C (2006) Bandit based Monte-Carlo planning. Proceedings of the 17th European Conference on Machine Learning, eds Fürnkranz J, Scheffer T, Spiliopoulou M ( Springer, Berlin), pp 282-293.
45. Kearns M, Mansour Y, Ng AY (2002) A sparse sampling algorithm for near-optimal planning in large Markov decision processes. Mach Learn 49(2-3):193-208.
46. Barto AG, Mahadevan S (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dyn Syst 13(4):341-379.
47. Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition 113(3):262-280.
48. Diuk C, Tsai K, Wallis J, Botvinick M, Niv Y (2013) Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia. J Neurosci 33(13): 5797-5805.
49. Ribas-Fernandes JJF, et al. (2011) A neural signature of hierarchical reinforcement learning. Neuron 71(2):370-379.
50. Solway A, et al. (2014) Optimal behavioral hierarchy. PLOS Comput Biol 10(8): e1003779.
51. Koller D, Friedman N (2009) Probabilistic Graphical Models: Principles and Techniques (MIT Press, Cambridge, MA).
52. Towal RB, Mormann M, Koch C (2013) Simultaneous modeling of visual saliency and value computation improves predictions of economic choice. Proc Natl Acad Sci USA 110(40):E3858-E3867.
53. Wimmer GE, Shohamy D (2012) Preference by association: How memory mechanisms in the hippocampus bias decisions. Science 338(6104):270-273.
54. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433-436.
55. Price K, Storn RM, Lampinen JA (2006) Differential Evolution: A Practical Approach to Global Optimization (Springer, New York).
56. Mullen KM, Ardia D, Gil DL, Windover D, Cline J (2009) DEoptim: An R package for global optimization by differential evolution. J Stat Softw 40(6):1-26.
57. R Core Team (2013) R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna).