Environmental statistics and the trade-off between model-based and TD learning in humans

Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, NIPS 2011

Volumn , Issue , 2011, Pages

  Simon, Dylan A ^a   Daw, Nathaniel D ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ECONOMIC AND SOCIAL EFFECTS; LEARNING SYSTEMS;

ENVIRONMENTAL STATISTICS; HIGH VOLATILITY; HIGH-LOW; MODEL FREE; MODEL-BASED OPC; MODELING-BASED LEARNING; RELATIVE PERFORMANCE; STATISTICAL EFFICIENCY; TD-LEARNING; TRADE OFF;

DECISION MAKING;

EID: 85162381627     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (24)

1. Bernard W. Balleine, Nathaniel D. Daw, and John P. O'Doherty. Multiple forms of value learning and the function of dopamine. In Paul W. Glimcher, Colin F. Camerer, Ernst Fehr, and Russell A. Poldrack, editors, Neuroeconomics: Decision Making and the Brain, chapter 24, pages 367-387. Academic Press, London, 2008.
2. Antoine Bechara. Decision making, impulse control and loss of willpower to resist drugs: a neurocognitive perspective. Nat Neurosci, 8(11):1458-63, 2005.
3. Frederick Toates. The interaction of cognitive and stimulus-response processes in the control of behaviour. Neuroscience & Biobehavioral Reviews, 22(1):59-83, 1997.
4. Peter Dayan. Goal-directed control and its antipodes. Neural Netw, 22:213-219, 2009.
5. Neal Schmitt, Bryan W. Coyle, and Larry King. Feedback and task predictability as determinants of performance in multiple cue probability learning tasks. Organ Behav Hum Perform, 16(2):388-402, 1976.
6. Berndt Brehmer and Jan Kuylenstierna. Task information and performance in probabilistic inference tasks. Organ Behav Hum Perform, 22:445-464, 1978.
7. B J Knowlton, L R Squire, and M A Gluck. Probabilistic classification learning in amnesia. Learn Mem, 1(2):106-120, 1994.
8. W. Todd Maddox and F. Gregory Ashby. Dissociating explicit and procedural-learning based systems of perceptual category learning. Behavioural Processes, 66(3):309-332, 2004.
9. W. Todd Maddox, J. Vincent Filoteo, Kelli D. Hejl, and A. David Ing. Category number impacts rule-based but not information-integration category learning: Further evidence for dissociable categorylearning systems. J Exp Psychol Learn Mem Cogn, 30(1):227-245, 2004.
10. R. A. Poldrack, J. Clark, E. J. Paré-Blagoev, D. Shohamy, J. Creso Moyano, C. Myers, and M. A. Gluck. Interactive memory systems in the human brain. Nature, 414(6863):546-550, 2001.
11. Bernard W. Balleine and Anthony Dickinson. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology, 37(4-5):407-419, 1998.
12. Kenji Doya. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw, 12(7-8):961-974, 1999.
13. Nathaniel D. Daw, Yael Niv, and Peter Dayan. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci, 8(12):1704-1711, 2005.
14. Ben Seymour, John P. O'Doherty, Peter Dayan, Martin Koltzenburg, Anthony K. Jones, Raymond J. Dolan, Karl J. Friston, and Richard S. Frackowiak. Temporal difference models describe higher-order learning in humans. Nature, 429(6992):664-667, 2004.
15. John P. O'Doherty, Peter Dayan, Johannes Schultz, Ralf Deichmann, Karl Friston, and Raymond J. Dolan. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669):452-454, 2004.
16. Adam Johnson and A. David Redish. Hippocampal replay contributes to within session learning in a temporal difference reinforcement learning model. Neural Netw, 18(9):1163-1171, 2005.
17. Máté Lengyel and Peter Dayan. Hippocampal contributions to control: The third way. In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 889-896. MIT Press, Cambridge, MA, 2008.
18. Mehdi Keramati, Amir Dezfouli, and Payam Piray. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol, 7(5):e1002055, 2011.
19. Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In Michael S. Kearns, Sara A. Solla, and David A. Cohn, editors, Advances in Neural Information Processing Systems 11, volume 11, pages 996-1002. MIT Press, Cambridge, MA, 1999.
20. R. E. Kalman. A new approach to linear filtering and prediction problems. J Basic Eng, 82(1):35-45, 1960.
21. Nathaniel D Daw, S. J. Gershman, B. Seymour, P. Dayan, and R. J. Dolan. Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6):1204-1215, 2011.
22. Brian Lau and Paul W Glimcher. Dynamic response-by-response models of matching behavior in rhesus monkeys. J Exp Anal Behav, 84(3):555-579, 2005.
23. Douglas Bates, Martin Maechler, and Ben Bolker. lme4: Linear mixed-effects models using S4 classes, 2011. R package version 0.999375-39.
24. Saori C Tanaka, Kazuyuki Samejima, Go Okada, Kazutaka Ueda, Yasumasa Okamoto, Shigeto Yamawaki, and Kenji Doya. Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw, 19(8):1233-1241, 2006.