Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia

Journal of Neuroscience

Volumn 33, Issue 13, 2013, Pages 5797-5805

  Diuk, Carlos ^a   Tsai, Karin ^b   Wallis, Jonathan ^c   Botvinick, Matthew ^a   Niv, Yael ^a  

^a PRINCETON UNIVERSITY   (United States)

^b Princeton University   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ANALYTICAL ERROR; ARTICLE; BASAL GANGLION; BEHAVIOR; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC NERVE CELL; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; OUTCOME ASSESSMENT; PREDICTION; PRIORITY JOURNAL; PROBABILITY; SPIKE; STATE DEPENDENT LEARNING; TASK PERFORMANCE; VENTRAL TEGMENTUM;

ADOLESCENT; ADULT; BASAL GANGLIA; BRAIN MAPPING; CHOICE BEHAVIOR; FEMALE; HIERARCHY, SOCIAL; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LEARNING; LOGISTIC MODELS; MAGNETIC RESONANCE IMAGING; MALE; NEURAL NETWORKS (COMPUTER); OXYGEN; REINFORCEMENT (PSYCHOLOGY); YOUNG ADULT;

EID: 84875468581     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.5445-12.2013     Document Type: Article

References (42)

1. Abler B, Walter H, Erk S, Kammerer H, Spitzer M (2006) Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 31:790-795. CrossRef Medline
2. Barto AG (1995) Adaptive critics and the basal ganglia. In: Models of information processing in the basal ganglia (Houk JC, Davis J, Beiser D, eds), pp 215-232. Cambridge, MA: MIT.
3. Barto AG, Mahadevan S (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 13:341-379. CrossRef
4. Behrens TE, Hunt LT, Woolrich MW, Rushworth MF. S (2008) Associative learning of social value. Nature 456:245-249. CrossRef Medline
5. Botvinick MM (2012) Hierarchical reinforcement learning and decision making. Curr Opin Neurobiol 22:956-962. CrossRef Medline
6. Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113:262-280. CrossRef Medline
7. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10:443-446. CrossRef Medline
8. Brischoux F, Chakraborty S, Brierley DI, Ungless MA (2009) Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A 106:4894-4899. CrossRef Medline
9. Burke CJ, Tobler PN, Baddeley M, Schultz W (2010) Neural mechanisms of observational learning. Proc Natl Acad Sci U S A 107:14431-14436. CrossRef Medline
10. Cohen JD, McClure SM, Yu AJ (2007) Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philos Trans R Soc Lond B Biol Sci 362:933-942. CrossRef Medline
11. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319:1264-1267. CrossRef Medline
12. Daw ND (2009) Trial-by-trial data analysis using computational models. Attention and Performance (1-26).
13. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441:876-879. CrossRef Medline
14. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron 69:1204-1215. CrossRef Medline
15. Dayan P (1993) Improving generalization for temporal difference learning: the successor representation. Neural Computation 5:613-624. CrossRef
16. Delgado MR, Miller MM, Inati S, Phelps EA (2005) An fMRI study of reward-related probability learning. Neuroimage 24: 862-73. CrossRef Medline
17. Dietterich TG (2000) Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research 13:227-303.
18. Diuk C, Botvinick MM, Barto AG, Niv Y (2010) Program No. 36:907.13. 2010 Neuroscience Meeting Planner, San Diego, CA: Society for Neuroscience.
19. Gershman SJ, Pesaran B, Daw ND (2009) Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J Neurosci 29:13524-13531. CrossRef Medline
20. Gläscher J, Daw N, Dayan P, O'Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66:585-595. CrossRef Medline
21. Glimcher PW (2011) Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc Natl Acad Sci U S A 108.
22. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28:5623-5630. CrossRef Medline
23. Hayes TR, Petrov AA, Sederberg PB (2011) A novel method for analyzing sequential eye movements reveals strategic influence on Raven's advanced progressive matrices. J Vis 11:10. CrossRef Medline
24. Klein-Flugge MC, Hunt LT, Bach DR, Dolan RJ, Behrens TE (2011) Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron 72:654-664. CrossRef Medline
25. Koller D, Parr R (1999) Computing factored value functions for policies in structured MDPs. In: Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence (Dean T, ed), pp 1332-1339. San Francisco: Morgan Kaufman.
26. Li J, McClure SM, King-Casas B, Montague PR (2006) Policy adjustment in a dynamic economic game. PLoS ONE 1:e103. CrossRef Medline
27. Li J, Daw ND (2011) Signals in human striatum are appropriate for policy update rather than value prediction. J Neurosci 31:5504-5511. CrossRef Medline
28. Lohrenz T, McCabe K, Camerer CF, Montague PR (2007) Neural signature of fictive learning signals in a sequential investment task. Proc Natl Acad Sci U S A 104:9493-9498. CrossRef Medline
29. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339-346. CrossRef Medline
30. Montague PR, Dayan P, Sejnowski TJ (1996) A Framework for Mesen-cephalic Predictive Hebbian Learning. J Neurosci 16:1936-1947. Medline
31. Niv Y, Edlund JA, Dayan P, O'Doherty JP (2012) Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain. J Neurosci 32:551-562. CrossRef Medline
32. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329-337. CrossRef Medline
33. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454. CrossRef Medline
34. Preuschoff K, Bossaerts P, Quartz SR (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron.
35. Ribas-Fernandes JJ, Solway A, Diuk C, McGuire JT, Barto AG, Niv Y, Botvinick MM (2011) A neural signature of hierarchical reinforcement learning. Neuron 71:370-379. CrossRef Medline
36. Schönberg T, Daw ND, Joel D, O'Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from non-learners during reward-based decision making. J Neurosci 27:12860-12867. CrossRef Medline
37. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36: 241-263. CrossRef Medline
38. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599. CrossRef Medline
39. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Cambridge, MA: MIT.
40. Sutton RS, Precup D, Singh S (1999) Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112:181-211. CrossRef
41. Suzuki S, Harasawa N, Ueno K, Gardner JL, Ichinohe N, Haruno M, Cheng K, Nakahara H (2012) Learning to simulate others' decisions. Neuron 74: 1125-1137. CrossRef Medline
42. Thorndike EL (1991) Animal intelligence: experimental studies. New York, NY: Macmillan.