Reward representations and reward-related learning in the human brain: Insights from neuroimaging

Current Opinion in Neurobiology

Volumn 14, Issue 6, 2004, Pages 769-776

  O'Doherty, John P ^a,b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

AMYGDALOID NUCLEUS; BRAIN; CODING; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC SYSTEM; LEARNING; MESENCEPHALON; NERVE CELL NETWORK; NEUROIMAGING; NONHUMAN; ORBITAL CORTEX; PREDICTION; PREFRONTAL CORTEX; PRIORITY JOURNAL; REVIEW; REWARD;

ANIMALS; BRAIN; BRAIN MAPPING; COGNITION; DECISION MAKING; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NERVE NET; NEURAL PATHWAYS; REWARD; TOMOGRAPHY;

EID: 9644310472     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.conb.2004.10.016     Document Type: Review

References (64)

1. E.T. Rolls, Z.J. Sienkiewicz, and S. Yaxley Hunger modulates the responses to gustatory stimuli of single neurons in the caudolateral orbitofrontal cortex of the macaque monkey Eur J Neurosci 1 1989 53 60
2. H.D. Critchley, and E.T. Rolls Hunger and satiety modify the responses of olfactory and visual neurons in the primate orbitofrontal cortex J Neurophysiol 75 1996 1673 1686
3. J. O'Doherty, E.T. Rolls, S. Francis, R. Bowtell, and F. McGlone Representation of pleasant and aversive taste in the human brain J Neurophysiol 85 2001 1315 1321
4. D.M. Small, M.D. Gregory, Y.E. Mak, D. Gitelman, M.M. Mesulam, and T. Parrish Dissociation of neural representation of intensity and affective valuation in human gustation Neuron 39 2003 701 711
5. J.A. Gottfried, R. Deichmann, J.S. Winston, and R.J. Dolan Functional heterogeneity in human olfactory cortex: an event-related functional magnetic resonance imaging study J Neurosci 22 2002 10819 10828
6. E.T. Rolls, M.L. Kringelbach, and I.E. De Araujo Different representations of pleasant and unpleasant odours in the human brain Eur J Neurosci 18 2003 695 703
7. A.K. Anderson, K. Christoff, I. Stappen, D. Panitz, D.G. Ghahremani, G. Glover, J.D. Gabrieli, and N. Sobel Dissociated neural representations of intensity and valence in human olfaction Nat Neurosci 6 2003 196 202 This fMRI study attempts to distinguish between brain areas processing valence and those processing intensity of odours, by matching some odours for intensity (although varying in valence) and others for valence (although varying in intensity). OFC is implicated in coding for valence, whereas the amygdala is proposed to be involved exclusively in processing odour intensity. This could suggest a primarily sensory rather than affective role for amygdala in stimulus processing. Interestingly, OFC responses in medial OFC were related to positive valence odours, whereas OFC responses in lateral OFC were related to negative valence odours.
8. E.T. Rolls, J. O'Doherty, M.L. Kringelbach, S. Francis, R. Bowtell, and F. McGlone Representations of pleasant and painful touch in the human orbitofrontal and cingulate cortices Cereb Cortex 13 2003 308 317
9. A.J. Blood, R.J. Zatorre, P. Bermudez, and A.C. Evans Emotional responses to pleasant and unpleasant music correlate with activity in paralimbic brain regions Nat Neurosci 2 1999 382 387
10. I. Aharon, N. Etcoff, D. Ariely, C.F. Chabris, E. O'Connor, and H.C. Breiter Beautiful faces have variable reward value: fMRI and behavioral evidence Neuron 32 2001 537 551
11. J. O'Doherty, J. Winston, H. Critchley, D. Perrett, D.M. Burt, and R.J. Dolan Beauty in a smile: the role of medial orbitofrontal cortex in facial attractiveness Neuropsychologia 41 2003 147 155
12. H. Kawabata, and S. Zeki Neural correlates of beauty J Neurophysiol 91 2004 1699 1705
13. R. Elliott, J.L. Newman, O.A. Longe, and J.F. Deakin Differential response patterns in the striatum and orbitofrontal cortex to financial reward in humans: a parametric functional magnetic resonance imaging study J Neurosci 23 2003 303 307
14. B.J. Rolls, E.T. Rolls, E.A. Rowe, and K. Sweeney Sensory specific satiety in man Physiol Behav 27 1981 137 142
15. J. O'Doherty, E.T. Rolls, S. Francis, R. Bowtell, F. McGlone, G. Kobal, B. Renner, and G. Ahne Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex Neuroreport 11 2000 893 897
16. M.L. Kringelbach, J. O'Doherty, E.T. Rolls, and C. Andrews Activation of the human orbitofrontal cortex to a liquid food stimulus is correlated with its subjective pleasantness Cereb Cortex 13 2003 1064 1071
17. J.S. Morris, C.D. Frith, D.I. Perrett, D. Rowland, A.W. Young, A.J. Calder, and R.J. Dolan A differential neural response in the human amygdala to fearful and happy facial expressions Nature 383 1996 812 815
18. D.H. Zald, and J.V. Pardo Emotion, olfaction, and the human amygdala: amygdala activation during aversive olfactory stimulation Proc Natl Acad Sci USA 94 1997 4119 4124
19. T. Canli, H. Sivers, S.L. Whitfield, I.H. Gotlib, and J.D. Gabrieli Amygdala response to happy faces as a function of extraversion Science 296 2002 2191
20. P.C. Holland, and M. Gallagher Amygdala-frontal interactions and reward expectancy Curr Opin Neurobiol 14 2004 148 155 This excellent review covers similar ground to the present review, but discusses in more detail the role of amygdala and OFC in reward prediction, encompassing recent findings from the animal literature as well as human neuroimaging.
21. R. Adolphs Neural systems for recognizing emotion Curr Opin Neurobiol 12 2002 169 177
22. Dayan P, Abbott L: Classical conditioning and reinforcement learning. In Theoretical Neuroscience. Cambridge, MA: MIT Press; 2001:331-358.
23. F.S. Arana, J.A. Parkinson, E. Hinton, A.J. Holland, A.M. Owen, and A.C. Roberts Dissociable contributions of the human amygdala and orbitofrontal cortex to incentive motivation and goal selection J Neurosci 23 2003 9632 9638 In this PET study subjects were scanned while perusing fictional restaurant menus containing either high or low incentive items (based on individual preferences). In one condition subjects had to choose specific items, whereas in another condition they merely had to read the menu. OFC activity was related to choice of menu items (in some instances modulated by incentive value), whereas amygdala responses were related to incentive value, but independent of behavioural choice.
24. C.R. Gallistel Conditioning from an information processing perspective Behav Processes 62 2003 89 101
25. B. Knutson, G.W. Fong, C.M. Adams, J.L. Varner, and D. Hommer Dissociation of reward anticipation and outcome with event-related fMRI Neuroreport 12 2001 3683 3687
26. J.P. O'Doherty, R. Deichmann, H.D. Critchley, and R.J. Dolan Neural responses during anticipation of a primary taste reward Neuron 33 2002 815 826
27. J.A. Gottfried, J. O'Doherty, and R.J. Dolan Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging J Neurosci 22 2002 10829 10837
28. J.A. Gottfried, J. O'Doherty, and R.J. Dolan Encoding predictive reward value in human amygdala and orbitofrontal cortex Science 301 2003 1104 1107
29. I.P. Pavlov Conditioned reflexes 1927 Oxford University Press Oxford
30. K. Zener The significance of behavior accompanying conditioned salivary secretion for theories of the conditioned response Am J Psychol 50 1937 384 403
31. B. Knutson, G.W. Fong, S.M. Bennett, C.M. Adams, and D. Hommer A region of mesial prefrontal cortex tracks monetarily rewarding outcomes: characterization with rapid event-related fMRI Neuroimage 18 2003 263 272 In this and previous studies from the same authors (e.g. [25]), a dissociation is proposed between regions involved in reward prediction and those in reward receipt. Whereas ventral striatum is implicated in reward prediction, medial prefrontal cortex is suggested to be involved in responding to receipt of the reward itself.
32. R.A. Rescorla, and A.R. Wagner A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement A.H. Black W.F. Prokasy Classical Conditioning II: Current Research and Theory 1972 Appleton Crofts New York 64 99
33. R.S. Sutton, and A.G. Barto Time derivative models of Pavlovian reinforcement M. Gabriel J. Moore Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990 MIT Press Cambridge, MA 497 537
34. W. Schultz, P. Dayan, and P.R. Montague A neural substrate of prediction and reward Science 275 1997 1593 1599
35. W. Schultz Predictive reward signal of dopamine neurons J Neurophysiol 80 1998 1 27
36. J.P. O'Doherty, P. Dayan, K. Friston, H. Critchley, and R.J. Dolan Temporal difference models and reward-related learning in the human brain Neuron 38 2003 329 337
37. S.M. McClure, G.S. Berns, and P.R. Montague Temporal prediction errors in a passive learning task activate human striatum Neuron 38 2003 339 346 In this study prediction errors are induced not by unexpected omission or delivery of reward, but rather by varying when in the trial the reward is delivered. Delivery of reward later in a trial than expected produced activity in the putamen (albeit more dorsally than in O'Doherty et al. [36]). Sensitivity to stimulus timing is a key feature of temporal difference models of classical conditioning.
38. D.H. Zald, I. Boileau, W. El Dearedy, R. Gunn, F. McGlone, G.S. Dichter, and A. Dagher Dopamine transmission in the human striatum during monetary reward tasks J Neurosci 24 2004 4105 4112 The authors use raclopride PET to demonstrate dopamine release in the human striatum during performance of a reward-learning task.
39. M.R. Delgado, L.E. Nystrom, C. Fissell, D.C. Noll, and J.A. Fiez Tracking the hemodynamic responses to reward and punishment in the striatum J Neurophysiol 84 2000 3072 3077
40. C.F. Zink, G. Pagnoni, M.E. Martin, M. Dhamala, and G.S. Berns Human striatal response to salient nonrewarding stimuli J Neurosci 23 2003 8092 8097
41. J.C. Horvitz Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events Neuroscience 96 2000 651 656
42. C.F. Zink, G. Pagnoni, M.E. Martin-Skurski, J.C. Chappelow, and G.S. Berns Human striatal responses to monetary reward depend on saliency Neuron 42 2004 509 517
43. L. Becerra, H.C. Breiter, R. Wise, R.G. Gonzalez, and D. Borsook Reward circuitry activation by noxious thermal stimuli Neuron 32 2001 927 946
44. J. Jensen, A.R. McIntosh, A.P. Crawley, D.J. Mikulis, G. Remington, and S. Kapur Direct activation of the ventral striatum in anticipation of aversive stimuli Neuron 40 2003 1251 1257 This study shows that ventral striatum is activated not only by predictors of reward but also by predictors of aversive stimuli, which implies that this region is not reward specific but has a more general role in motivational processing.
45. G. Pagnoni, C.F. Zink, P.R. Montague, and G.S. Berns Activity in human ventral striatum locked to errors of reward prediction Nat Neurosci 5 2002 97 98
46. B. Seymour, J.P. O'Doherty, P. Dayan, A.K. Jones, R.J. Dolan, K.J. Friston, and R.S. Frackowiak Temporal difference models describe higher-order learning for pain Nature 429 2004 664 667 The authors demonstrate that temporal difference prediction error signals are present in ventral striatum not only during reward prediction but also during prediction of aversive stimuli (pain). Furthermore, prediction errors are shown following unexpected delivery or omission of learned cues rather than following delivery or omission of the pain itself.
47. W. Schultz, L. Tremblay, and J.R. Hollerman Changes in behavior-related neuronal activity in the striatum during learning Trends Neurosci 26 2003 321 328
48. M. Haruno, T. Kuroda, K. Doya, K. Toyama, M. Kimura, K. Samejima, H. Imamizu, and M. Kawato A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task J Neurosci 24 2004 1660 1665 This study shows that during action selection for reward, activity in dorsal striatum correlates with a measure of learning as assessed by the degree of convergence between subjects' responses and optimal behaviour in the task.
49. R. Elliott, J.L. Newman, O.A. Longe, and J.F. William Deakin Instrumental responding for rewards is associated with enhanced neuronal response in subcortical reward systems Neuroimage 21 2004 984 990
50. J. O'Doherty, P. Dayan, J. Schultz, R. Deichmann, K. Friston, and R.J. Dolan Dissociable roles of ventral and dorsal striatum in instrumental conditioning Science 304 2004 452 454
51. E.M. Tricomi, M.R. Delgado, and J.A. Fiez Modulation of caudate activity by action contingency Neuron 41 2004 281 292 The authors show that dorsal striatum activity occurs during action selection even if there is just a perceived contingency between action selection and reward (in actuality rewards were delivered non-contingently).
52. P.R. Montague, P. Dayan, and T.J. Sejnowski A framework for mesencephalic dopamine systems based on predictive Hebbian learning J Neurosci 16 1996 1936 1947
53. N. Ramnani, and R.C. Miall Instructed delay activity in the human prefrontal cortex is modulated by monetary reward expectation Cereb Cortex 13 2003 318 327
54. J. O'Doherty, H. Critchley, R. Deichmann, and R.J. Dolan Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices J Neurosci 23 2003 7931 7939
55. R.D. Rogers, N. Ramnani, C. Mackay, J.L. Wilson, P. Jezzard, C.S. Carter, and S.M. Smith Distinct portions of anterior cingulate cortex and medial prefrontal cortex are activated by reward processing in separable phases of decision-making cognition Biol Psychiatry 55 2004 594 602
56. A. Bechara, D. Tranel, and H. Damasio Characterization of the decision-making deficit of patients with ventromedial prefrontal cortex lesions Brain 123 2000 2189 2202
57. M.K. Sanghera, E.T. Rolls, and A. Roper-Hall Visual responses of neurons in the dorsolateral amygdala of the alert monkey Exp Neurol 63 1979 610 626
58. J.S. Morris, and R.J. Dolan Dissociable amygdala and orbitofrontal responses during reversal fear conditioning Neuroimage 22 2004 372 380
59. M.G. Baxter, A. Parker, C.C. Lindner, A.D. Izquierdo, and E.A. Murray Control of response selection by reinforcer value requires interaction of amygdala and orbital prefrontal cortex J Neurosci 20 2000 4311 4319
60. K.J. Friston, L. Harrison, and W. Penny Dynamic causal modelling Neuroimage 19 2003 1273 1302
61. M. Watanabe, K. Hikosaka, M. Sakagami, and S. Shirakawa Coding and monitoring of motivational context in the primate prefrontal cortex J Neurosci 22 2002 2391 2400
62. J. Tanji, K. Shima, and Y. Matsuzaka Reward-based planning of motor selection in the rostral cingulate motor area Adv Exp Med Biol 508 2002 417 423
63. A.N. McCoy, J.C. Crowley, G. Haghighian, H.L. Dean, and M.L. Platt Saccade reward signals in posterior cingulate cortex Neuron 40 2003 1031 1040
64. M.L. Platt, and P.W. Glimcher Neural correlates of decision variables in parietal cortex Nature 400 1999 233 238