Cerebral correlates of salient prediction error for different rewards and punishments

Cerebral Cortex

Volumn 23, Issue 2, 2013, Pages 477-487

  Metereau, Elise ^a,b   Dreher, Jean Claude ^a,b  

^a INSTITUT DES SCIENCES COGNITIVES   (France)

^b UNIVERSITÉ LYON 1   (France)

Author keywords

appetitive conditioning; aversive conditioning; classical conditioning; fMRI; reinforcement learning model

Indexed keywords

ADULT; AMYGDALOID NUCLEUS; ANTERIOR CINGULATE; ANTERIOR INSULA; APPETITE; ARTICLE; BRAIN CORTEX; CORPUS STRIATUM; FEMALE; FUNCTIONAL MAGNETIC RESONANCE IMAGING; HUMAN; HUMAN EXPERIMENT; LEARNING; MALE; NORMAL HUMAN; PREDICTION; PRIORITY JOURNAL; PUNISHMENT; REINFORCEMENT; REWARD; REWARD PREDICTION ERROR; SALIENT PREDICTION ERROR; SYSTEMATIC ERROR;

BRAIN; BRAIN MAPPING; CONDITIONING, CLASSICAL; FEMALE; HUMANS; LEARNING; MAGNETIC RESONANCE IMAGING; MALE; PUNISHMENT; REWARD;

EID: 84864269599     PISSN: 10473211     EISSN: 14602199     Source Type: Journal    
DOI: 10.1093/cercor/bhs037     Document Type: Article

References (97)

1. Abler B, Walter H, Erk S, Kammerer H, Spitzer M. 2006. Prediction error as a linear function of reward probability is coded in human nucleus accumbens. Neuroimage. 31(2):790-795.
2. Bayer HM, Glimcher PW. 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron. 47(1):129-141.
3. Bechara A, Damasio H, Damasio AR. 2003. Role of the amygdala in decision-making. Ann N Y Acad Sci. 985:356-369.
4. Behrens TEJ, Hunt LT, Woolrich MW, Rushworth MFS. 2008. Associative learning of social value. Nature. 456(7219):245-249.
5. Belova MA, Paton JJ, Salzman CD. 2008. Moment-to-moment tracking of state value in the amygdala. J Neurosci. 28(40):10023-10030.
6. Berns GS, McClure SM, Pagnoni G, Montague PR. 2001. Predictability modulates human brain response to reward. J Neurosci. 21(8): 2793-2798.
7. Berridge KC, Robinson TE. 1998. What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev. 28(3):309-369.
8. Bjork JM, Hommer DW. 2007. Anticipating instrumentally obtained and passively-received rewards: A factorial fMRI investigation. Behav Brain Res. 177(1):165-170.
9. Bray S, O'Doherty J. 2007. Neural coding of reward-prediction error signals during classical conditioning with attractive faces. J Neurophysiol. 97(4):3036-3045.
10. Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P. 2001. Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron. 30(2):619-639.
11. Brischoux F, Chakraborty S, Brierley DI, Ungless MA. 2009. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A. 106(12):4894-4899.
12. Bromberg-Martin ES, Matsumoto M, Hikosaka O. 2010. Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron. 67(1):144-155.
13. Buchel C, Dolan RJ, Armony JL, Friston KJ. 1999. Amygdala-hippocampal involvement in human aversive trace conditioning revealed through event-related functional magnetic resonance imaging. J Neurosci. 19(24):10869-10876.
14. Carter RM, Macinnes JJ, Huettel SA, Adcock RA. 2009. Activation in the VTA and nucleus accumbens increases in anticipation of both gains and losses. Front Behav Neurosci. 3:21.
15. Cohen MX, Ranganath C. 2005. Behavioral and neural predictors of upcoming decisions. Cogn Affect Behav Neurosci. 5(2): 117-126.
16. Cooper JC, Knutson B. 2008. Valence and salience contribute to nucleus accumbens activation. Neuroimage. 39(1):538-547.
17. Critchley HD, Mathias CJ, Dolan RJ. 2002. Fear conditioning in humans: The influence of awareness and autonomic arousal on functional neuroanatomy. Neuron. 33(4):653-663.
18. D'Ardenne K, McClure SM, Nystrom LE, Cohen JD. 2008. BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science. 319(5867):1264-1267.
19. Daw ND, Kakade S, Dayan P. 2002. Opponent interactions between serotonin and dopamine. Neural Netw. 15(4-6):603-616.
20. Delgado MR, Li J, Schiller D, Phelps EA. 2008. The role of the striatum in aversive learning and aversive prediction errors. Philos Trans R Soc Lond B Biol Sci. 363(1511):3787-3800.
21. Delgado MR, Nystrom LE, Fissell C, Noll DC, Fiez JA. 2000. Tracking the hemodynamic responses to reward and punishment in the striatum. J Neurophysiol. 84(6):3072-3077.
22. Dreher J-C, Grafman J. 2002. The roles of the cerebellum and basal ganglia in timing and error prediction. Eur J Neurosci. 16(8):1609-1619.
23. Dreher J-C, Kohn P, Berman KF. 2006. Neural coding of distinct statistical properties of reward information in humans. Cereb Cortex. 16(4):561-573.
24. Eickhoff SB, Stephan KE, Mohlberg H, Grefkes C, Fink GR, Amunts K, Zilles K. 2005. A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage. 25(4):1325-1335.
25. Gottfried JA, Deichmann R, Winston JS, Dolan RJ. 2002. Functional heterogeneity in human olfactory cortex: An event-related functional magnetic resonance imaging study. J Neurosci. 22(24): 10819-10828.
26. Gottfried JA, O'Doherty J, Dolan RJ. 2002. Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. J Neurosci. 22(24):10829-10837.
27. Gottfried JA, O'Doherty J, Dolan RJ. 2003. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science. 301(5636):1104-1107.
28. Hammers A, Allom R, Koepp MJ, Free SL, Myers R, Lemieux L, Mitchell TN, Brooks DJ, Duncan JS. 2003. Three-dimensional maximum probability atlas of the human brain, with particular reference to the temporal lobe. Hum Brain Mapp. 19(4):224-247.
29. Hare TA, O'Doherty J, Camerer CF, Schultz W, Rangel A. 2008. Dissociating the role of the orbitofrontal cortex and the striatum in the computation of goal values and prediction errors. J Neurosci. 28(22):5623-5630.
30. Herwig U, Abler B, Walter H, Erk S. 2007. Expecting unpleasant stimuli - An fMRI study. Psychiatry Res. 154(1):1-12.
31. Horvitz JC. 2000. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience. 96(4):651-656.
32. Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer CF. 2005. Neural systems responding to degrees of uncertainty in human decision-making. Science. 310(5754):1680-1683.
33. Ide JS, Li C-SR. 2011. Error-related functional connectivity of the habenula in humans. Front Hum Neurosci. 5:25.
34. Jensen J, McIntosh AR, Crawley AP, Mikulis DJ, Remington G, Kapur S. 2003. Direct activation of the ventral striatum in anticipation of aversive stimuli. Neuron. 40(6):1251-1257.
35. Jensen J, Smith AJ, Willeit M, Crawley AP, Mikulis DJ, Vitcu I, Kapur S. 2007. Separate brain regions code for salience vs. valence during reward prediction in humans. Hum Brain Mapp. 28(4):294-302.
36. Kahn I, Yeshurun Y, Rotshtein P, Fried I, Ben-Bashat D, Hendler T. 2002. The role of the amygdala in signaling prospective outcome of choice. Neuron. 33(6):983-994.
37. Knight DC, Waters NS, King MK, Bandettini PA. 2010. Learning-related diminution of unconditioned SCR and fMRI signal responses. Neuroimage. 49(1):843-848.
38. Knutson B, Adams CM, Fong GW, Hommer D. 2001. Anticipation of increasing monetary reward selectively recruits nucleus accumbens. J Neurosci. 21(16):RC159.
39. Knutson B, Westdorp A, Kaiser E, Hommer D. 2000. FMRI visualization of brain activity during a monetary incentive delay task. Neuroimage. 12(1):20-27.
40. Kringelbach ML, O'Doherty J, Rolls ET, Andrews C. 2003. Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness. Cereb Cortex. 13(10):1064-1071.
41. LaBar KS, Gatenby JC, Gore JC, LeDoux JE, Phelps EA. 1998. Human amygdala activation during conditioned fear acquisition and extinction: A mixed-trial fMRI study. Neuron. 20(5):937-945.
42. Lang PJ, Bradley MM, Cuthbert BN. 2005. International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Gainesville (FL): University of Florida. No. Technical Report A-6.
43. LeDoux JE. 2000. Emotion circuits in the brain. Ann Rev Neurosci. 23:155-184.
44. Li C-sR, Chao HHA, Lee T-W. 2009. Neural correlates of speeded as compared with delayed responses in a stop signal task: An indirect analog of risk taking and association with an anxiety trait. Cereb Cortex. 19(4):839-848.
45. Li C-sR, Huang C, Constable RT, Sinha R. 2006. Imaging response inhibition in a stop-signal task: Neural correlates independent of signal monitoring and post-response processing. J Neurosci. 26(1):186-192.
46. Liu X, Powell DK, Wang H, Gold BT, Corbly CR, Joseph JE. 2007. Functional dissociation in frontal and striatal areas for processing of positive and negative reward information. J Neurosci. 27(17):4587-4597.
47. Machado CJ, Kazama AM, Bachevalier J. 2009. Impact of amygdala, orbital frontal, or hippocampal lesions on threat avoidance and emotional reactivity in nonhuman primates. Emotion. 9(2):147-163.
48. Matsumoto M, Hikosaka O. 2007. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature. 447(7148):1111-1115.
49. Matsumoto M, Hikosaka O. 2009a. Representation of negative motivational value in the primate lateral habenula. Nat Neurosci. 12(1):77-84.
50. Matsumoto M, Hikosaka O. 2009b. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature. 459(7248):837-841.
51. Matsumoto M, Matsumoto K, Abe H, Tanaka K. 2007. Medial prefrontal cell activity signaling prediction errors of action values. Nat Neurosci. 10(5):647-656.
52. Mazaika P, Hoeft F, Glover G, Reiss AL. 2009. Methods and software for fMRI analysis for clinical subjects. Presentation at the 15th Annual Meeting of the Organization for Human Brain Mapping San Francisco, CA, June 18-23, 2009 spnl.stanford.edu
53. McClure SM, Berns GS, Montague PR. 2003. Temporal prediction errors in a passive learning task activate human striatum. Neuron. 38(2):339-346.
54. Menon M, Jensen J, Vitcu I, Graff-Guerrero A, Crawley A, Smith MA, Kapur S. 2007. Temporal difference modeling of the blood-oxygen level dependent response during aversive conditioning in humans: Effects of dopaminergic modulation. Biol Psychiatry. 62(7):765-772.
55. Miyazaki K, Miyazaki KW, Doya K. 2012. The role of serotonin in the regulation of patience and impulsivity. Mol Neurobiol. doi: 10.1007/ s12035-012-8232-6.
56. Murray EA. 2007. The amygdala, reward and emotion. Trends Cogn Sci. 11(11):489-497.
57. Nichols T, Brett M, Andersson J, Wager T, Poline J-B. 2005. Valid conjunction inference with the minimum statistic. Neuroimage. 25(3):653-660.
58. Nieuwenhuis S, Heslenfeld DJ, Geusau NJAv, Mars RB, Holroyd CB, Yeung N. 2005. Activity in human reward-sensitive brain areas is strongly context dependent. Neuroimage. 25(4):1302-1309.
59. O'Doherty J, Critchley H, Deichmann R, Dolan RJ. 2003. Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices. J Neurosci. 23(21):7931-7939.
60. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ. 2004. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science. 304(5669):452-454.
61. O'Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ. 2003. Temporal difference models and reward-related learning in the human brain. Neuron. 38(2):329-337.
62. Oldfield RC. 1971. The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia. 9(1):97-113.
63. Pan W-X, Schmidt R, Wickens JR, Hyland BI. 2005. Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network. J Neurosci. 25(26):6235-6242.
64. Paton JJ, Belova MA, Morrison SE, Salzman CD. 2006. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature. 439(7078):865-870.
65. Pessiglione M, Seymour B, Flandin G, Dolan RJ, Frith CD. 2006. Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature. 442(7106):1042-1045.
66. Pezze MA, Feldon J. 2004. Mesolimbic dopaminergic pathways in fear conditioning. Prog Neurobiol. 74(5):301-320.
67. Ravel S, Legallet E, Apicella P. 2003. Responses of tonically active neurons in the monkey striatum discriminate between motivationally opposing stimuli. J Neurosci. 23(24):8489-8497.
68. Redgrave P, Prescott TJ, Gurney K. 1999. Is the short-latency dopamine response too short to signal reward error? Trends Neurosci. 22(4):146-151.
69. Rescorla RA, Wagner AR. 1972. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black A, Prokasy WF, editors. Classical conditioning II: Current research and theory. New York: Appleton- Century-Crofts. p. 64-99.
70. Reynolds JN, Hyland BI, Wickens Jr. 2001. A cellular mechanism of reward-related learning. Nature. 413(6851):67-70.
71. Roesch MR, Calu DJ, Esber GR, Schoenbaum G. 2010. Neural correlates of variations in event processing during learning in basolateral amygdala. J Neurosci. 30(7):2464-2471.
72. Roesch MR, Calu DJ, Schoenbaum G. 2007. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci. 10(12):1615-1624.
73. Rutledge RB, Dean M, Caplin A, Glimcher PW. 2010. Testing the reward prediction error hypothesis with an axiomatic model. J Neurosci. 30(40):13525-13536.
74. Salas R, Baldwin P, de Biasi M, Montague PR. 2010. BOLD responses to negative reward prediction errors in human habenula. Front Hum Neurosci. 4:36.
75. Sander D, Grafman J, Zalla T. 2003. The human amygdala: An evolved system for relevance detection. Rev Neurosci. 14(4):303-316.
76. Sarinopoulos I, Grupe DW, Mackiewicz KL, Herrington JD, Lor M, Steege EE, Nitschke JB. 2010. Uncertainty during anticipation modulates neural responses to aversion in human insula and amygdala. Cereb Cortex. 20(4):929-940.
77. Schultz W. 1998. Predictive reward signal of dopamine neurons. J Neurophysiol. 80(1):1-27.
78. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science. 275(5306):1593-1599.
79. Sescousse G, Redoute J, Dreher J-C. 2010. The architecture of reward value coding in the human orbitofrontal cortex. J Neurosci. 30(39):13095-13104.
80. Seymour B, Daw N, Dayan P, Singer T, Dolan R. 2007. Differential encoding of losses and gains in the human striatum. J Neurosci. 27(18):4826-4831.
81. Seymour B, O'Doherty JP, Dayan P, Koltzenburg M, Jones AK, Dolan RJ, Friston KJ, Frackowiak RS. 2004. Temporal difference models describe higher-order learning in humans. Nature. 429(6992):664-667.
82. Seymour B, O'Doherty JP, Koltzenburg M, Wiech K, Frackowiak R, Friston K, Dolan R. 2005. Opponent appetitive-Aversive neural processes underlie predictive learning of pain relief. Nat Neurosci. 8(9):1234-1240.
83. Skinner BF. 1938. The behavior of organisms: An experimental analysis. New York: Appleton-Century-Crofts
84. Small DM, Gregory MD, Mak YE, Gitelman D, Mesulam MM, Parrish T. 2003. Dissociation of neural representation of intensity and affective valuation in human gustation. Neuron. 39(4):701-711.
85. Small DM, Zatorre RJ, Dagher A, Evans AC, Jones-Gotman M. 2001. Changes in brain activity related to eating chocolate: From pleasure to aversion. Brain. 124(Pt 9):1720-1733.
86. Stefanacci L, Amaral DG. 2002. Some observations on cortical inputs to the macaque monkey amygdala: An anterograde tracing study. J Comp Neurol. 451(4):301-323.
87. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S. 2004. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci. 7(8):887-893.
88. Tobler PN, O'Doherty JP, Dolan RJ, Schultz W. 2007. Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J Neurophysiol. 97(2):1621-1632.
89. Tom SM, Fox CR, Trepel C, Poldrack RA. 2007. The neural basis of loss aversion in decision-making under risk. Science. 315(5811):515-518.
90. Tremblay L, Schultz W. 1999. Relative reward preference in primate orbitofrontal cortex. Nature. 398(6729):704-708.
91. Tricomi EM, Delgado MR, Fiez JA. 2004. Modulation of caudate activity by action contingency. Neuron. 41(2):281-292.
92. Wise RA. 2004. Dopamine, learning and motivation. Nature Reviews. Neuroscience. 5(6):483-494.
93. Yacubian J, Gläscher J, Schroeder K, Sommer T, Braus DF, Büchel C. 2006. Dissociable systems for gain- And loss-related value predictions and errors of prediction in the human brain. J Neurosci. 26(37):9530-9537.
94. Young AMJ. 2004. Increased extracellular dopamine in nucleus accumbens in response to unconditioned and conditioned aversive stimuli: Studies using 1 min microdialysis in rats. J Neurosci Methods. 138(1-2):57-63.
95. Zaghloul KA, Blanco JA, Weidemann CT, McGill K, Jaggi JL, Baltuch GH, Kahana MJ. 2009. Human substantia nigra neurons encode unexpected financial rewards. Science. 323(5920):1496-1499.
96. Zink CF, Pagnoni G, Chappelow J, Martin-Skurski M, Berns GS. 2006. Human striatal activation reflects degree of stimulus saliency. Neuroimage. 29(3):977-983.
97. Zink CF, Pagnoni G, Martin-Skurski ME, Chappelow JC, Berns GS. 2004. Human striatal responses to monetary reward depend on saliency. Neuron. 42(3):509-517.