Scaling prediction errors to reward variability benefits error-driven learning in humans

Journal of Neurophysiology

Volumn 114, Issue 3, 2015, Pages 1628-1640

  Diederen, Kelly M J ^a   Schultz, Wolfram ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Adaptation; Probability distribution; Reinforcement learning; Risk; Standard deviation

Indexed keywords

ACCURACY; ADAPTATION; ADULT; ARTICLE; ASSOCIATION; CONTROLLED STUDY; FEMALE; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; MEASUREMENT ERROR; NORMAL HUMAN; PREDICTION; PRIORITY JOURNAL; PROBABILITY; RATING SCALE; REINFORCEMENT; REWARD; STANDARD; TASK PERFORMANCE; ADOLESCENT; ANTICIPATION; BIOLOGICAL MODEL; STATISTICAL MODEL;

ADOLESCENT; ADULT; ANTICIPATION, PSYCHOLOGICAL; FEMALE; HUMANS; KNOWLEDGE OF RESULTS (PSYCHOLOGY); MALE; MODELS, NEUROLOGICAL; MODELS, STATISTICAL; REWARD;

EID: 84941243691     PISSN: 00223077     EISSN: 15221598     Source Type: Journal    
DOI: 10.1152/jn.00483.2015     Document Type: Article

References (28)

1. Behrens TE, Woolrich MW, Walton ME, Rushworth MF. Learning the value of information in an uncertain world. Nat Neurosci 10: 1214–1221, 2007.
2. Berniker M, Voss M, Kording K. Learning priors for Bayesian computations in the nervous system. PloS One 5: e12686, 2010.
3. Cox RT. Probability, frequency, and reasonable expectation. Am J Phys 14: 1–13, 1946.
4. Garrison J, Erdeniz B, Done J. Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies. Neurosci Biobehav Rev 37: 1297–1310, 2013.
5. Jaynes ET. Bayesian Methods: General Background. Cambridge, UK: Cambridge Univ. Press, 1986.
6. Koechlin E, Hyafil A. Anterior prefrontal function and the limits of human decision-making. Science 318: 594–598, 2007.
7. Kording KP, Wolpert DM. Bayesian integration in sensorimotor learning. Nature 427: 244–247, 2004.
8. Krugel LK, Biele G, Mohr PN, Li SC, Heekeren HR. Genetic variation in dopaminergic neuromodulation influences the ability to rapidly and flexibly adapt decisions. Proc Natl Acad Sci USA 106: 17951–17956, 2009.
9. Lee SW, Shimojo S, O’Doherty JP. Neural computations underlying arbitration between model-based and model-free learning. Neuron 81: 687–699, 2014.
10. LePelley ME, McLaren IP. Associative history affects the associative change undergone by both presented and absent cues in human causal learning. J Exp Psychol Anim Behav Process 30: 67–73, 2004.
11. Lewandowsky S, Farrell S. Computational Modeling in Cognition: Principles and Practice. Thousand Oaks, CA: Sage, 2011.
12. Li J, Schiller D, Schoenbaum G, Phelps EA, Daw ND. Differential roles of human striatum and amygdala in associative learning. Nat Neurosci 14: 1250–1252, 2011.
13. Nassar MR, Wilson RC, Heasly B, Gold JI. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J Neurosci 30: 12366–12378, 2010.
14. O’Reilly JX. Making predictions in a changing world-inference, uncertainty, and learning. Front Neurosci 7: 105, 2013.
15. O’Reilly JX, Jbabdi S, Behrens TE. How can a Bayesian approach inform neuroscience? Eur J Neurosci 35: 1169–1179, 2012.
16. Park SQ, Kahnt T, Talmi D, Rieskamp J, Dolan RJ, Heekeren HR. Adaptive coding of reward prediction errors is gated by striatal coupling. Proc Natl Acad Sci USA 109: 4285–4289, 2012.
17. Payzan-LeNestour E, Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput Biol 7: e1001048, 2011.
18. Payzan-LeNestour E, Dunne S, Bossaerts P, O’Doherty JP. The neural representation of unexpected uncertainty during value-based decision making. Neuron 79: 191–201, 2013.
19. Pearce JM, Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev 87: 532–552, 1980.
20. Preuschoff K, Bossaerts P. Adding prediction risk to the theory of reward learning. Ann NY Acad Sci 1104: 135–146, 2007.
21. Rescorla RA, Wagner AR. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical Conditioning II: Current Research and Theory, edited by BlackAH, Prokasy WF. New York: Appleton Century Crofts, 1972, p. 64-99.
22. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science 275: 1593–1599, 1997.
23. Schultz W, Dickinson A. Neuronal coding of prediction errors. Annu Rev Neurosci 23: 473–500, 2000.
24. Stocker AA, Simoncelli EP. Noise characteristics and prior expectations in human visual speed perception. Nat Neurosci 9: 578–585, 2006.
25. Tobler PN, Fiorillo CD, Schultz W. Adaptive coding of reward value by dopamine neurons. Science 307: 1642–1645, 2005.
26. Vilares I, Howard JD, Fernandes HL, Gottfried JA, Kording KP. Differential representations of prior and likelihood uncertainty in the human brain. Curr Biol 22: 1641–1648, 2012.
27. Yu AJ, Dayan P. Uncertainty, neuromodulation, attention. Neuron 46: 681– 692, 2005.
28. Yuille A, Kersten D. Vision as Bayesian inference: analysis by synthesis? Trends Cogn Sci 10: 301–308, 2006.