The problem with value

Neuroscience and Biobehavioral Reviews

Volumn 43, Issue , 2014, Pages 259-268

  O'Doherty, John P ^a  

^a CALIFORNIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Decision making; Learning; Neuroeconomics; Reward

Indexed keywords

DOPAMINE;

AMYGDALOID NUCLEUS; ANALYTICAL ERROR; ASSOCIATION; BEHAVIOR; BOLD SIGNAL; BRAIN FUNCTION; BRAIN REGION; HUMAN; MUSCULOSKELETAL FUNCTION; NERVE POTENTIAL; NERVE STIMULATION; NEUROANATOMY; NEUROBIOLOGY; NEUROPHYSIOLOGY; NEUROSCIENCE; NONHUMAN; NOTE; ORBITAL CORTEX; PREDICTION; PREDICTIVE VALUE; PRIORITY JOURNAL; PROBABILITY; SIGNAL TRANSDUCTION; ANIMAL; BIOLOGICAL MODEL; BRAIN; DECISION MAKING; LEARNING; PHYSIOLOGY; REWARD;

ANIMALS; BRAIN; DECISION MAKING; HUMANS; LEARNING; MODELS, NEUROLOGICAL; PREDICTIVE VALUE OF TESTS; REWARD;

EID: 84901837691     PISSN: 01497634     EISSN: 18737528     Source Type: Journal    
DOI: 10.1016/j.neubiorev.2014.03.027     Document Type: Note

References (91)

1. Balleine B.W., Daw N.D., O'Doherty J.P. Multiple forms of value learning and the function of dopamine. Neuroeconomics: Decision Making and the Brain 2008, 367-385. Elsevier, New York. P.W. Glimcher, C. Camerer, E. Fehr, R.A. Poldrack (Eds.).
2. Balleine B.W., Dickinson A. Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 1998, 37:407-419.
3. Balleine B.W., Killcross S. Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci. 2006, 29:272-279.
4. Baxter M.G., Parker A., Lindner C.C., Izquierdo A.D., Murray E.A. Control of response selection by reinforcer value requires interaction of amygdala and orbital prefrontal cortex. J. Neurosci. 2000, 20:4311-4319.
5. Bechara A., Damasio A.R., Damasio H., Anderson S.W. Insensitivity to future consequences following damage to human prefrontal cortex. Cognition 1994, 50:7-15.
6. Behrens T.E., Woolrich M.W., Walton M.E., Rushworth M.F. Learning the value of information in an uncertain world. Nat. Neurosci. 2007, 10:1214-1221.
7. Bentham J. An Introduction to the Principles of Morals and Legislation (Dover Philosophical Classics) 2007, Dover Publications, New York.
8. Berridge K.C. Food reward: brain substrates of wanting and liking. Neurosci. Biobehav. Rev. 1996, 20:1-25.
9. Berridge K.C. From prediction error to incentive salience: mesolimbic computation of reward motivation. Eur. J. Neurosci. 2012, 35:1124-1143.
10. Brown P.L., Jenkins H.M. Auto-shaping of the pigeon's key-peck. J. Exp. Anal. Behav. 1968, 11:1-8.
11. Cacioppo J.T., Berntson G.G., Larsen J.T., Poehlmann K.M., Ito T.A. The psychophysiology of emotion. Handb. Emotion. 2000, 2:173-191.
12. Camerer C.F. Neuroeconomics: opening the gray box. Neuron 2008, 60:416-419.
13. Chib V.S., Rangel A., Shimojo S., O'Doherty J.P. Evidence for a common representation of decision values for dissimilar goods in human ventromedial prefrontal cortex. J. Neurosci. 2009, 29:12315-12320.
14. Colwill R.M., Motzkin D.K. Encoding of the unconditioned stimulus in Pavlovian conditioning. Anim. Learn. Behav. 1994, 22:384-394.
15. Cools R., Clark L., Owen A.M., Robbins T.W. Defining the neural mechanisms of probabilistic reversal learning using event-related functional magnetic resonance imaging. J. Neurosci. 2002, 22:4563-4567.
16. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8:1704-1711.
17. Dean M. Chapter 7: what can neuroeconomics tell us about economics (and vice versa)?. Comparative Decision-Making 2013, Oxford University Press, New York. T.R. Zentall, P.H. Crowley (Eds.).
18. Dickinson A. Actions and habits: the development of a behavioural autonomy. Philos. Trans. R. Soc. Lond. B Biol. Sci. 1985, 308:67-78.
19. Everitt B.J., Cardinal R.N., Parkinson J.A., Robbins T.W. Appetitive behavior: impact of amygdala-dependent mechanisms of emotional learning. Ann. N. Y. Acad. Sci. 2003, 985:233-250.
20. Fehr E., Camerer C.F. Social neuroeconomics: the neural circuitry of social preferences. Trends Cogn. Sci. 2007, 11:419-427.
21. Fellows L.K., Farah M.J. Different underlying impairments in decision-making following ventromedial and dorsolateral frontal lobe damage in humans. Cereb. Cortex 2005, 15:58-63.
22. Glimcher P.W., Rustichini A. Neuroeconomics: the consilience of brain and decision. Science 2004, 306:447-452.
23. Gottfried J.A., O'Doherty J., Dolan R.J. Appetitive and aversive olfactory learning in humans studied using event-related functional magnetic resonance imaging. J. Neurosci. 2002, 22:10829-10837.
24. Gottfried J.A., O'Doherty J., Dolan R.J. Encoding predictive reward value in human amygdala and orbitofrontal cortex. Science 2003, 301:1104-1107.
25. Grabenhorst F., Rolls E.T. Value, pleasure and choice in the ventral prefrontal cortex. Trends Cogn. Sci. 2011, 15:56-67.
26. Gray M.A., Harrison N.A., Wiens S., Critchley H.D. Modulation of emotional appraisal by false physiological feedback during fMRI. PLoS One 2007, 2:e546.
27. Hampshire A., Chaudhry A.M., Owen A.M., Roberts A.C. Dissociable roles for lateral orbitofrontal cortex and lateral prefrontal cortex during preference driven reversal learning. NeuroImage 2012, 59:4102-4112.
28. Hampton A.N., Bossaerts P., O'Doherty J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 2006, 26:8360-8367.
29. Holland P.C., Straub J.J. Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 1979, 5:65-78.
30. Hollerman J.R., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1998, 1:304-309.
31. Hornak J., O'Doherty J., Bramham J., Rolls E.T., Morris R.G., Bullock P.R., Polkey C.E. Reward-related reversal learning after surgical excisions in orbito-frontal or dorsolateral prefrontal cortex in humans. J. Cogn. Neurosci. 2004, 16:463-478.
32. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 2000, 96:651-656.
33. Jarvik M.E., Kopp R. An improved one-trial passive avoidance learning situation. Psychol. Rep. 1967, 21:221-224.
34. Jenkins H.M., Moore B.R. The form of the auto-shaped response with food or water reinforcers. J. Exp. Anal. Behav. 1973, 20:163-181.
35. Kable J.W., Glimcher P.W. The neural correlates of subjective value during intertemporal choice. Nat. Neurosci. 2007, 10:1625-1633.
36. Kahneman D., Wakker P.P., Sarin R. Back to Bentham? Explorations of experienced utility. Q. J. Econ. 1997, 112:375-406.
37. Klein-Flugge M.C., Barron H.C., Brodersen K.H., Dolan R.J., Behrens T.E. Segregated encoding of reward-identity and stimulus-reward associations in human orbitofrontal cortex. J. Neurosci. 2013, 33:3202-3211.
38. Konorski J. Conditioned Reflexes and Neuron Organization 1948.
39. Kreibig S.D. Autonomic nervous system activity in emotion: a review. Biol. Psychol. 2010, 84:394-421.
40. Lak A., Stauffer W.R., Schultz W. Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc. Natl. Acad. Sci. U.S.A. 2014, 111:2343-2348.
41. Lang P.J., Greenwald M.K., Bradley M.M., Hamm A.O. Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology 1993, 30:261-273.
42. Lau B., Glimcher P.W. Action and outcome encoding in the primate caudate nucleus. J. Neurosci. 2007, 27:14502-14514.
43. Leathers M.L., Olson C.R. In monkeys making value-based decisions, LIP neurons encode cue salience and not action value. Science 2012, 338:132-135.
44. Levy I., Snell J., Nelson A.J., Rustichini A., Glimcher P.W. Neural representation of subjective value under risk and ambiguity. J. Neurophysiol. 2010, 103:1036-1047.
45. Mackintosh N.J. A theory of attention: variations in the associability of stimuli with reinforcement. Psychol. Rev. 1975, 82:276-298.
46. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
47. Maunsell J.H. Neuronal representations of cognitive state: reward or attention?. Trends Cogn. Sci. 2004, 8:261-265.
48. McDannald M.A., Jones J.L., Takahashi Y.K., Schoenbaum G. Learning theory: a driving force in understanding orbitofrontal function. Neurobiol. Learn. Mem. 2014, 108C:22-27.
49. McNamee D., Rangel A., O'Doherty J.P. Category-dependent and category-independent goal-value codes in human ventromedial prefrontal cortex. Nat. Neurosci. 2013, 16:479-485.
50. Mirenowicz J., Schultz W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 1994, 72:1024-1027.
51. Montague P.R., Berns G.S. Neural economics and the biological substrates of valuation. Neuron 2002, 36:265-284.
52. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9:1057-1063.
53. Noonan M.P., Walton M.E., Behrens T.E., Sallet J., Buckley M.J., Rushworth M.F. Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex. Proc. Natl. Acad. Sci. U.S.A. 2010, 107:20547-20552.
54. O'Doherty J., Critchley H., Deichmann R., Dolan R.J. Dissociating valence of outcome from behavioral control in human orbital and ventral prefrontal cortices. J. Neurosci. 2003, 23:7931-7939.
55. O'Doherty J., Kringelbach M.L., Rolls E.T., Hornak J., Andrews C. Abstract reward and punishment representations in the human orbitofrontal cortex. Nat. Neurosci. 2001, 4:95-102.
56. O'Doherty J.P. Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr. Opin. Neurobiol. 2004, 14:769-776.
57. O'Doherty J.P. Lights, camembert, action! The role of human orbitofrontal cortex in encoding stimuli, rewards, and choices. Ann. N. Y. Acad. Sci. 2007, 1121:254-272.
58. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441:223-226.
59. Paton J.J., Belova M.A., Morrison S.E., Salzman C.D. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 2006, 439:865-870.
60. Pavlov I.P. Conditioned Reflexes 1927, Oxford University Press, Oxford.
61. Payzan-LeNestour E., Bossaerts P. Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings. PLoS Comput. Biol. 2011, 7:e1001048.
62. Payzan-LeNestour E., Dunne S., Bossaerts P., O'Doherty J.P. The neural representation of unexpected uncertainty during value-based decision making. Neuron 2013, 79:191-201.
63. Pearce J.M., Hall G. A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 1980, 87:532-552.
64. Pickens C.L., Saddoris M.P., Setlow B., Gallagher M., Holland P.C., Schoenbaum G. Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task. J. Neurosci. 2003, 23:11078-11084.
65. Plassmann H., O'Doherty J., Rangel A. Orbitofrontal cortex encodes willingness to pay in everyday economic transactions. J. Neurosci. 2007, 27:9984-9988.
66. Platt M.L., Glimcher P.W. Neural correlates of decision variables in parietal cortex. Nature 1999, 400:233-238.
67. Rangel A., Camerer C., Montague P.R. A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 2008, 9:545-556.
68. Roesch M.R., Calu D.J., Esber G.R., Schoenbaum G. All that glitters . . . dissociating attention and outcome expectancy from prediction errors signals. J. Neurophysiol. 2010, 104:587-595.
69. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 2007, 10:1615-1624.
70. Roesch M.R., Esber G.R., Li J., Daw N.D., Schoenbaum G. Surprise! Neural correlates of Pearce-Hall and Rescorla-Wagner coexist within the brain. Eur. J. Neurosci. 2012, 35:1190-1200.
71. Roesch M.R., Olson C.R. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 2004, 304:307-310.
72. Rolls B.J., Rolls E.T., Rowe E.A., Sweeney K. Sensory specific satiety in man. Physiol. Behav. 1981, 27:137-142.
73. Rolls E.T. Emotion Explained 2007, Oxford University Press, Oxford.
74. Rolls E.T., Hornak J., Wade D., McGrath J. Emotion-related learning in patients with social and emotional changes associated with frontal lobe damage. J. Neurol. Neurosurg. Psychiatry 1994, 57:1518-1524.
75. Samejima K., Ueda Y., Doya K., Kimura M. Representation of action-specific reward values in the striatum. Science 2005, 310:1337-1340.
76. Sanfey A.G., Loewenstein G., McClure S.M., Cohen J.D. Neuroeconomics: cross-currents in research on decision-making. Trends Cogn. Sci. 2006, 10:108-116.
77. Schoenbaum G., Chiba A.A., Gallagher M. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci. 1998, 1:155-159.
78. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 1998, 80:1-27.
79. Shapiro M.G., Frazier S.J., Lester H.A. Unparalleled control of neural activity using orthogonal pharmacogenetics. ACS Chem. Neurosci. 2012, 3:619-629.
80. Sohn J.W., Lee D. Order-dependent modulation of directional signals in the supplementary and presupplementary motor areas. J. Neurosci. 2007, 27:13655-13666.
81. Solomon R.L., Corbit J.D. An opponent-process theory of motivation: I. Temporal dynamics of affect. Psychol. Rev. 1974, 81:119-145.
82. Sugrue L.P., Corrado G.S., Newsome W.T. Matching behavior and the representation of value in the parietal cortex. Science 2004, 304:1782-1787.
83. Thorndike E.L. Animal Intelligence: An Experimental Study of the Associative Processes in Animals 1898, Macmillan, New York.
84. Tremblay L., Schultz W. Relative reward preference in primate orbitofrontal cortex. Nature 1999, 398:704-708.
85. Valentin V.V., Dickinson A., O'Doherty J.P. Determining the neural substrates of goal-directed learning in the human brain. J. Neurosci. 2007, 27:4019-4026.
86. Williams S.C., Deisseroth K. Optogenetics. Proc. Natl. Acad. Sci. U.S.A. 2013, 110:16287.
87. Wunderlich K., Rangel A., O'Doherty J.P. Neural computations underlying action-based decision making in the human brain. Proc. Natl. Acad. Sci. U.S.A. 2009, 106:17199-17204.
88. Wunderlich K., Rangel A., O'Doherty J.P. Economic choices can be made using only stimulus values. Proc. Natl. Acad. Sci. U.S.A. 2010, 107:15005-15010.
89. Yin H.H., Knowlton B.J., Balleine B.W. Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci. 2004, 19:181-189.
90. Yin H.H., Ostlund S.B., Knowlton B.J., Balleine B.W. The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci. 2005, 22:513-523.
91. Zink C.F., Pagnoni G., Chappelow J., Martin-Skurski M., Berns G.S. Human striatal activation reflects degree of stimulus saliency. NeuroImage 2006, 29:977-983.