Testing the reward prediction error hypothesis with an axiomatic model

Journal of Neuroscience

Volumn 30, Issue 40, 2010, Pages 13525-13536

  Rutledge, Robb B ^a   Dean, Mark ^a   Caplin, Andrew ^a   Glimcher, Paul W ^a  

^a NEW YORK UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

OXYGEN;

ADULT; AMYGDALOID NUCLEUS; ARTICLE; CAUDATE NUCLEUS; CINGULATE GYRUS; COGNITION; CORPUS STRIATUM; ERROR; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; HYPOTHESIS; INSULA; MALE; MODEL; NERVE CELL; NORMAL HUMAN; NUCLEUS ACCUMBENS; OXYGEN BLOOD LEVEL; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; PUTAMEN; REWARD; THALAMUS; THEORY;

ADULT; ALGORITHMS; BEHAVIOR; BRAIN MAPPING; COGNITION; FEMALE; FORECASTING; FUNCTIONAL LATERALITY; HUMANS; IMAGE PROCESSING, COMPUTER-ASSISTED; LEARNING; MALE; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; PREDICTIVE VALUE OF TESTS; PROSENCEPHALON; REGRESSION ANALYSIS; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 77957728784     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.1747-10.2010     Document Type: Article

References (52)

1. Abler B, Walter H, Erk S, Kammerer H, Spitzer M (2006) Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 31:790-795. (Pubitemid 43776671)
2. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141.
3. Behrens TEJ, Hunt LT, Woolrich MW, Rushworth MFS (2008) Associative learning of social value. Nature 456:245-249.
4. Belova MA, Paton JJ, Morrison SE, Salzman CD (2007) Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron 55:970-984.
5. Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Rev 28:309-369.
6. Brischoux F, Chakraborty S, Brierley DI, Ungless MA (2009) Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A 106:4894-4899.
7. Caplin A, Dean M (2008a) Dopamine, reward prediction error, and economics. Q J Econ 123:663-701.
8. Caplin A, Dean M (2008b) Axiomatic methods, dopamine and reward prediction error. Curr Opin Neurobiol 18:197-202.
9. Caplin A, Dean M, Glimcher PW, Rutledge RB (2010) Measuring beliefs and rewards: a neuroeconomic approach. Q J Econ 125:923-960.
10. Caviness VSJ, Meyer J, Makris N, Kennedy DN (1996) MRI-based topographic parcellation of the human neocortex: an anatomically specified method with estimate of reliability. J Cogn Neurosci 8:566-587. (Pubitemid 27075437)
11. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319:1264-1267. (Pubitemid 351323026)
12. Day JJ, Roitman MF, Wightman RM, Carelli RM (2007) Associate learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat Neurosci 10:1020-1028.
13. de Bruijn ERA, de Lange FP, von Cramon DY, Ullsperger M (2009) When errors are rewarding. J Neurosci 29:12183-12186.
14. Friston KJ, Josephs O, Rees G, Turner R (1998) Nonlinear event-related respones in fMRI. Magn Reson Med 39:41-52.
15. Grinband J, Hirsch J, Ferrera VP (2006) A neural representation of categorization uncertainty in the human brain. Neuron 49:757-763. (Pubitemid 43290134)
16. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of orbitofrontal cortex and striatum in the computation of goal values and prediction errors. J Neurosci 28:5623-5630.
17. Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nat Neurosci 1:304-309.
18. Horvitz JC (2000) Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96:651-656.
19. Huettel SA, Song AW, McCarthy G (2005) Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. J Neurosci 25:3304-3311.
20. Jensen J, Smith AJ, Willeit M, Crawley AP, Mikulis DJ, Vitcu I, Kapur S (2007) Separate brain regions code for salience vs. valence during reward prediction in humans. Hum Brain Mapp 28:294-302. (Pubitemid 46542911)
21. Joshua M, Adler A, Mitelman R, Vaadia E, Bergman H (2008) Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J Neurosci 28:11673-11684.
22. Li J, McClure SM, King-Casas B, Montague PR (2006) Policy adjustment in a dynamic economic game. PloS ONE 1:e103.
23. Matsumoto M, Hikosaka O (2007) Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447:1111-1115. (Pubitemid 47014424)
24. Matsumoto M, Hikosaka O (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837-841.
25. Matsumoto M, Matsumoto K, Abe H, Tanaka K (2007) Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci 10:647-656. (Pubitemid 46652446)
26. McClure SM, Berns GS, Montague PR (2003) Temporal predication errors in a passive learning task activate human striatum. Neuron 38:339-346.
27. McCoy AN, Crowley JC, Haghighian G, Dean HL, Platt ML (2003) Saccade reward signals in posterior cingulate cortex. Neuron 40:1031-1040. (Pubitemid 37522060)
28. Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O (2004) Dopamine neurons can represent context-dependent prediction error. Neuron 41:269-280. (Pubitemid 38125187)
29. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329-337. (Pubitemid 36555657)
30. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454. (Pubitemid 38495943)
31. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD (2006) Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442:1042-1045. (Pubitemid 44330261)
32. Phillips PEM, Stuber GD, Heien MLAV, Wightman RM, Carelli RM (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614-618. (Pubitemid 36460414)
33. Popper K (1959) The logic of scientific discovery. New York: Basic Books.
34. Preuschoff K, Bossaerts P, Quartz SR (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51:381-390. (Pubitemid 44160540)
35. Preuschoff K, Quartz SR, Bossaerts P (2008) Human insula activation reflects risk prediction errors as well as risk. J Neurosci 28:2745-2752. (Pubitemid 351398874)
36. Rademacher J, Galaburda AM, Kennedy DN, Filipek PA, Caviness VS Jr (1992) Human cerebral cortex: localization, parcellation, and morphometry with magnetic resonance imaging. J Cogn Neurosci 4:352-374.
37. Redgrave P, Prescott TJ, Gurney K (1999) Is the short-latency dopamine response too short to signal reward error? Trends Neurosci 22:146-151.
38. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II: Current research and theory (Black AH, Prokasy WF, eds), pp 64-99. New York: Appleton.
39. Rutledge RB, Lazzaro SC, Lau B, Myers CE, Gluck MA, Glimcher PW (2009) Dopaminergic drugs modulate learning rates and perseveration in Parkinson's patients in a dynamic foraging task. J Neurosci 29:15104-15114.
40. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599.
41. Seeley WW, Menon V, Schatzberg AF, Keller J, Glover GH, Kenna H, Reiss AL, Greicius MD (2007) Dissociable intrinsic connectivity networks for salience processing and executive control. J Neurosci 27:2349-2356. (Pubitemid 46340968)
42. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS (2004) Temporal difference models describe higher-order learning in humans. Nature 429:664-667. (Pubitemid 38812318)
43. Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. In: Learning and computational neuroscience: foundations of adaptive networks (Gabriel M, Moore J, eds), pp 497-537. Cambridge, MA: MIT.
44. Ullsperger M, von Cramon DY (2003) Error monitoring using external feedback: specific roles of the habenular complex, the reward system, and the cingulate motor area revealed by functional magnetic resonance imaging. J Neurosci 23:4308-4314. (Pubitemid 36958451)
45. Vazquez AL, Noll DC (1998) Nonlinear aspects of the BOLD response in functional MRI. Neuroimage 7:108-118.
46. Voon V, Pessiglione M, Brezing C, Gallea C, Fernandez HH, Dolan RJ, Hallett M (2010) Mechanisms underlying dopamine-mediated reward bias in compulsive behaviors. Neuron 65:135-142.
47. Wald A (1943) Tests of statistical hypotheses concerning several parameters when the number of observations is large. Trans Am Math Soc 54:426-482.
48. Wittmann BC, Daw ND, Seymour B, Dolan RJ (2008) Striatal activity underlies novelty-based choice in humans. Neuron 58:967-973. (Pubitemid 351842752)
49. Yacubian J, Gläscher J, Schroeder K, Sommer T, Braus DF, Büchel C (2006) Dissociable systems for gain-and loss-related value predictions and errors of prediction in the human brain. J Neurosci 26:9530-9537. (Pubitemid 44396866)
50. Zaghloul KA, Blanco JA, Weidemann CT, McGill K, Jaggi JL, Baltuch GH, Kahana MJ (2009) Human substantia nigra neurons encode unexpected financial rewards. Science 323:1496-1499.
51. Zink CF, Pagnoni G, Martin ME, Dhamala M, Berns GS (2003) Human striatal response to salient nonrewarding stimuli. J Neurosci 23:8092-8097. (Pubitemid 37082462)
52. Zink CF, Pagnoni G, Martin-Skurski ME, Chappelow JC, Berns GS (2004) Human striatal responses to monetary reward depend on saliency. Neuron 42:509-517. (Pubitemid 38610148)