Neural prediction errors reveal a risk-sensitive reinforcement-learning process in the human brain

Journal of Neuroscience

Volumn 32, Issue 2, 2012, Pages 551-562

  Niv, Yael ^a   Edlund, Jeffrey A ^b   Dayan, Peter ^c   O'Doherty, John P ^b,d  

^a PRINCETON UNIVERSITY   (United States)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

^d TRINITY COLLEGE DUBLIN   (Ireland)

Author keywords

[No Author keywords available]

Indexed keywords

ADULT; ARTICLE; BEHAVIOR; BOLD SIGNAL; DECISION MAKING; ERROR; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; MALE; MODEL; NORMAL HUMAN; NUCLEUS ACCUMBENS; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REINFORCEMENT LEARNING; REWARD; RISK; RISK SENSITIVITY; STIMULUS;

ADOLESCENT; BRAIN; CHOICE BEHAVIOR; CONDITIONING, CLASSICAL; FEMALE; HUMANS; LEARNING; MALE; MODELS, PSYCHOLOGICAL; NUCLEUS ACCUMBENS; PREDICTIVE VALUE OF TESTS; REINFORCEMENT (PSYCHOLOGY); REWARD; RISK-TAKING; YOUNG ADULT;

EID: 84855688852     PISSN: 02706474     EISSN: 15292401     Source Type: Journal    
DOI: 10.1523/JNEUROSCI.5498-10.2012     Document Type: Article

References (70)

1. Abler B, Walter H, Erk S, Kammerer H, Spitzer M (2006) Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage 31:790-795
2. Barto AG (1995) Adaptive critic and the basal ganglia. In: Models of information processing in the basal ganglia (Houk JC, Davis JL, Beiser DG, eds), pp 215-232. Cambridge, MA: MIT.
3. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141
4. Bernoulli D (1954) Exposition of a new theory on the measurement of risk. Econometrica 22:23-36
5. Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P (2001) Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30:619-639
6. Cox RW (1996) AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29:162-173
7. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science 319:1264-1267
8. Daw ND (2011) Trial by trial data analysis using computational models. In: Decision making, affect, and learning: attention and performance XXIII (Delgado MR, Phelps EA, Robbins TW, eds), pp 3-38
9. Oxford UK, Oxford UP, Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704-1711
10. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ (2006) Cortical substrates for exploratory decisions in humans. Nature 441:876-879
11. Dayan P, Kakade S, Montague PR (2000) Learning and selective attention. Nat Neurosci [Suppl] 3:1218-1223
12. Denrell J (2007) Adaptive learning and risk taking. Psychol Rev 114:177-187
13. Denrell J, March JG (2001) Adaptation as information restriction: the hot stove effect. Organ Sci 12:523-538
14. Elliott R, Rees G, Dolan RJ (1999) Ventromedial prefrontal cortex mediates guessing. Neuropsychologia 37:403-411
15. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-1902
16. Fitzgerald TH, Seymour B, Bach DR, Dolan RJ (2010) Differentiable neural substrates for learned and described value and risk. Curr Biol 20:1823-1829
17. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A (2008) Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci 28:5623-5630
18. Hayden B, Platt M (2008) Gambling for Gatorade: risk-sensitive decision making for fluid rewards in humans. Anim Cogn 12:201-207
19. Hertwig R, Erev I (2009) The description-experience gap in risky choice. Trends Cogn Sci 13:517-523
20. Hertwig R, Barron G, Weber EU, Erev I (2004) Decisions from experience and the effect of rare events in risky choice. Psychol Sci 15:534-539
21. Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer CF (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science 310:1680-1683
22. Hsu M, Krajbich I, Zhao C, Camerer CF (2009) Neural responses to reward anticipation under risk is nonlinear in probabilities. J Neurosci 29:2231-2237
23. Huettel SA, Song AW, McCarthy G (2005) Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. J Neurosci 25:3304-3311
24. Huettel SA, Stowe CJ, Gordon EM, Warner BT, Platt ML (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron 49:765-775
25. Jessup RK, Bishara AJ, Busemeyer JR (2008) Feedback produces divergence from prospect theory in descriptive choice. Psychol Sci 19:1015-1022
26. Kacelnik A, Bateson M (1996) Risky theories-the effect of variance on foraging decisions. Am Zoologist 36:402-434
27. Kim H, Shimojo S, O'Doherty JP (2006) Is avoiding an aversive outcome rewarding? neural substrates of avoidance learning in the human brain. PLoS Biol 4:e233.
28. Knutson B, Bossaerts P (2007) Neural antecedents of financial decisions. J Neurosci 27:8174-8177
29. Knutson B, Gibbs SE (2007) Linking nucleus accumbens dopamine and blood oxygenation. Psychopharmacology (Berl) 191:813-822
30. Knutson B, Taylor J, Kaufman M, Peterson R, Glover G (2005) Distributed neural representation of expected value. J Neurosci 25:4806-4812
31. Kuhnen CM, Knutson B (2005) The neural basis of financial risk taking. Neuron 47:763-770
32. Li J, McClure SM, King-Casas B, Montague PR (2006) Policy adjustment in a dynamic economic game. PLoS One 1:e103.
33. March JG (1996) Learning to be risk averse. Psychol Rev 103:309-319
34. Markowitz H (1952) Portfolio selection. J Finance 7:77-91
35. Matthews SC, Simmons AN, Lane SD, Paulus MP (2004) Selective activation of the nucleus accumbens during risk-taking decision making. Neuroreport 15:2123-2127
36. McClure SM, Berns GS, Montague PR (2003) Temporal prediction errors in a passive learning task activate human striatum. Neuron 38:339-346
37. McCoy AN, Platt ML (2005) Risk-sensitive neurons in macaque posterior cingulate cortex. Nat Neurosci 8:1220-1227
38. Mihatsch O, Neuneier R (2002) Risk-sensitive reinforcement learning. Machine Learn 49:267-290
39. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936-1947
40. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9:1057-1063
41. Niv Y (2009) Reinforcement learning in the brain. J Math Psychol 53:139-154
42. Niv Y, Schoenbaum G (2008) Dialogues on prediction errors. Trends Cogn Sci 12:265-272
43. Niv Y, Joel D, Meilijson I, Ruppin E (2002) Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviors. Adapt Behav 10:5-24
44. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty and TD learning. Behav Brain Funct 1:6.
45. O'Doherty J, Dayan P, Friston K, Critchley H, Dolan R (2003) Temporal difference learning model accounts for responses in human ventral striatum and orbitofrontal cortex during Pavlovian appetitive learning. Neuron 38:329-337
46. O'Doherty JP, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452-454
47. Platt ML, Huettel SA (2008) Risky business: the neuroeconomics of decision making under uncertainty. Nat Neurosci 11:398-403
48. Preuschoff K, Bossaerts P, Quartz SR (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51:381-390
49. Preuschoff K, Quartz SR, Bossaerts P (2008) Human insula activation reflects risk prediction errors as well as risk. J Neurosci 28:2745-2752
50. Rabin M, Thaler RH (2001) Anomalies: risk aversion. J Econom Perspect 15:219-232
51. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II: current research and theory (Black AH, Prokasy WF, eds), pp 64-99 New York: Appleton-Century-Crofts
52. Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10:1615-1624
53. Rouder JN, Speckman PL, Sun D, Morey RD, Iverson G (2009) Bayesian t tests for accepting and rejecting the null hypothesis. Psychon Bull Rev 16:225-237
54. Schönberg T, Daw ND, Joel D, O'Doherty JP (2007) Reinforcement learning signals in the human striatum distinguish learners from nonlearners during reward-based decision making. J Neurosci 27:12860-12867
55. Schott BH, Minuzzi L, Krebs RM, Elmenhorst D, Lang M, Winz OH, Seidenbecher CI, Coenen HH, Heinze H-J, Zilles K, Düzel E, Bauer A (2008) Mesolimbic functional magnetic resonance imaging activations during reward anticipation correlate with reward-related ventral striatal dopamine release. J Neurosci 28:14311-14319
56. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599
57. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS (2004) Temporal difference models describe higher-order learning in humans. Nature 429:664-667
58. Seymour B, Daw N, Dayan P, Singer T, Dolan R (2007) Differential encoding of losses and gains in the human striatum. J Neurosci 27:4826-4831
59. Shapiro MS, Couvillon PA, Bitterman ME (2001) Quantitative tests of an associative theory of risk-sensitivity in honeybees. J Exp Biol 204:565-573
60. Shen W, Flajolet M, Greengard P, Surmeier DJ (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321:848-851
61. Smallwood PD (1996) An introduction to risk sensitivity: the use of Jensen's inequality to clarify evolutionary arguments of adaptation and constraint. Am Zoologist 36:392-401
62. Sutton RS (1988) Learning to predict by the method of temporal difference. Machine Learning 3:9-44
63. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. Cambridge, MA: MIT.
64. Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307:1642-1645
65. Tobler PN, O'Doherty JP, Dolan RJ, Schultz W (2006) Human neural learning depends on reward prediction errors in the blocking paradigm. J Neurophysiol 95:301-310
66. Tobler PN, O'Doherty JP, Dolan RJ, Schultz W (2007) Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J Neurophysiol 97:1621-1632
67. Tom SM, Fox CR, Trepel C, Poldrack RA (2007) The neural basis of loss aversion in decision-making under risk. Science 315:515-518
68. Weber EU, Johnson EJ (2008) Decisions under uncertainty: psychological, economic, and neuroeconomic explanations of risk preference. In: Neuroeconomics: Decision making and the brain (Glimcher PW, Camerer C, Fehr E, Poldrack R, eds), pp 127-144 London: Elsevier
69. Weber EU, Shafir S, Blais, A-R (2004) Predicting risk sensitivity in humans and lower animals: risk as variance or coefficient of variation. Psychol Rev 111:430-445
70. Wickens JR, Reynolds JN, Hyland BI (2003) Neural mechanisms of rewardrelated motor learning. Curr Opin Neurobiol 13:685-690.