Choosing the greater of two goods: Neural currencies for valuation and decision making

Nature Reviews Neuroscience

Volumn 6, Issue 5, 2005, Pages 363-375

  Sugrue, Leo P ^a   Corrado, Greg S ^a   Newsome, William T ^a  

^a STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL BEHAVIOR; BRAIN; DECISION MAKING; FRONTAL CORTEX; NEUROBIOLOGY; NONHUMAN; PARIETAL LOBE; PRIMATE; PRIORITY JOURNAL; PROBABILITY; REVIEW;

ANIMALS; CHOICE BEHAVIOR; DECISION MAKING; HUMANS; NERVE NET; SOCIAL VALUES;

ANIMALIA; PRIMATES;

EID: 17844396920     PISSN: 1471003X     EISSN: None     Source Type: Journal    
DOI: 10.1038/nrn1666     Document Type: Review

References (107)

1. Bauby, J. D. The Diving Bel and The Butterfly: A Memoir of Life in Death (Alfred A. Knopf Inc., New York, 1997).
2. Glimcher, P. W. & Rustichini, A. R. Neuroeconomics: the consilience of brain and decision. Science 306, 447-452 (2004).
3. Parker, A. J. & Newsome, W. T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227-277 (1998). An exploration of the link between neural activity and sensory perception, summarizing work that comprises the background to recent neurophysiological studies of perceptual decision making.
4. Rizzolatti, G. & Luppino, G. The cortical motor system. Neuron 31, 889-901 (2001).
5. Graziano, M. S., Taylor, C. S., Moore, T. & Cooke, D. F. The cortical control of movement revisited. Neuron 36, 349-362 (2002).
6. Andersen, R. A. & Bruneo, C. A. Intentional maps in posterior parietal cortex. Annu. Rev. Neurosci. 25, 189-220 (2002).
7. Leon, M. I. & Shadlen, M. N. Exploring the neurophysiology of decisions. Neuron 21, 669-672 (1998).
8. Georgopoulos, A. P. Neural aspects of cognitive motor control. Curr. Opin. Neurobiol. 10, 238 (2000).
9. Schall, J. D. Neural basis of deciding, choosing and acting. Nature Rev. Neurosci. 2, 33-42 (2001).
10. Gold, J. I. & Shadlen, M. N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10-16 (2001). Proposes that neural systems implement categorical decisions about perceptual stimuli by computing a decision variable related to the logarithm of the likelihood ratio in favour of one or another alternative. Discusses how this framework can be used to interpret neural signals recorded from eye movement planning centres during perceptual discrimination tasks.
11. Glimcher, P. W. Making choices: the neurophysiology of visual-saccadic decision making. Trends Neurosci. 24, 654-659 (2001).
12. Romo, R. & Salinas, E, Touch and go: decision making mechanisms in somatosensation. Annu. Rev. Neurosci. 24, 107-137 (2001).
13. Romo, R., Hernandez, A., Zainos, A., Lemus, L. & Brody, C. Neural correlates of decision-making in secondary somato-sensory cortex. Nature Neurosci. 5, 1217-1225 (2002).
14. Romo, R. & Salinas, E. Flutter discrimination: neural codes, perception, memory and decision making. Nature Rev. Neurosci. 4, 203-218 (2003).
15. Graham, N. V. S. Visual Pattern Analysers (Oxford Univ. Press, Oxford, 1989).
16. Shadlen, M. N. & Newsome, W. T. Motion perception: seeing and deciding. Proc. Natl Acad. Sci. USA 93, 628-633 (1996). One of the first demonstrations of a correlation between the activity of single cortical neurons and an evolving perceptual decision.
17. Shadlen, M. N. & Newsome, W. T. Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol. 86, 1916-1936 (2001).
18. Kim, J. N. & Shadlen, M. N. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nature Neurosci. 2, 176-185 (1999).
19. Horwitz, G. D. & Newsome, W. T. Target selection for saccadic eye movements: prelude activity in the superior colliculus during a direction-discrimination task. J. Neurophysiol. 86, 2543-2558 (2001).
20. Roitman, J. D. & Shadlen, M. N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475-9489 (2002).
21. Hanks, T. D. & Shadlen, M. N. Microstimulation of macaque area UP affects decision making in a motion discrimination task. Soc. Neurosci. Abstr. 20.9 (2004).
22. Bisley, J. W. & Goldberg, M. E. Neuronal activity in the lateral intraparietal area and spatial attention. Science 299, 81-86 (2003).
23. Platt, M. L. & Glimcher, P. W. Neural correlates of decision variables in parietal cortex. Nature 400, 233-238 (1998).
24. Kahneman, D. & Tversky, A. (eds) Choices, Values, and Frames (Cambridge Univ. Press, Cambridge, 2000).
25. Berridge, K. C. & Robinson, T. E. Parsing reward. Trends Neurosci. 26, 507-513 (2003). Discusses the distinct psychological components of reward and their underlying neural substrates. Emphasizes the distinction between systems devoted to the affective ('liking') and motivational ('wanting') aspects of reward, and between motivational processes based on simple Pavlovian associations and those involving more cognitive representations of value.
26. Olds, J. & Milner, P. M. Positive reinforcement produced by electrical stimulation of septal area and other regions of the rat brain. J. Comp. Physiol. Psychol. 47, 419-427 (1954) .
27. Shizgal, P. Neural basis of utility estimation. Curr. Opin. Neurobiol. 7, 198-208 (1997). Summarizes findings from a series of elegant experiments involving BSR in the rat that indicate the existence of a final common neural representation for reward information.
28. Wise, R. A. & Bozarth, M. A. A psychomotor stimulant theory of addiction. Psychol. Rev. 94, 469-492 (1987).
29. Hyman, S. E. & Malenka, R. C. Addiction and the brain: the neurobiology of compulsion and Its persistence. Nature Rev. Neurosci. 2, 695-703 (2001).
30. Kelley, A. E. Memory and addiction. Shared neural circuitry and molecular mechanisms. Neuron 44, 161-167 (2004).
31. Berridge, K. C. & Robinson, T. E. What is the role of dopaminein reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309-369 (1998).
32. Garris, P A. et al. Dissociation of dopamine release in the nucleus accumbens from intracranial self-stimulation. Nature 398, 67-69 (1999).
33. Pecina, S., Cagniard, B., Berridge, K. C., Aldridge, J. W. & Zhuang, X. Hyperdopaminergic mutant mice have higher 'wanting' but not 'liking' for sweet rewards. J. Neurosci. 23, 9395-9402 (2003).
34. Liu, Z. et al. DNA targeting of rhinal cortex D2 receptor protein reversibly blocks learning of cues that predict reward. Proc. Natl Acad. Sci. USA 101, 12336-12341 (2004).
35. Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900-913 (1993).
36. Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27 (1998).
37. Hollerman, J. R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neurosci. 1, 304-309 (1998).
38. Sutton, R. S. & Barto, A. G. Reinforcement Learning (MIT Press, Cambridge, Massachusetts, 1998).
39. Dayan, P. & Abbott, L. F. Theoretical Neuroscience Ch. 9 (MIT Press, Cambridge, Massachusetts, 2001).
40. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593-1599 (1997). Reviews evidence indicating that midbrain dopamine neurons signal an error in the prediction of future reward (see references 35-37). Proposes that the function of these neurons is particularly well described by a specific class of reinforcement learning algorithms, and shows how a model that uses a dopamine-like signal to implement such an algorithm can learn to predict future rewards and guide action selection.
41. Schultz, W. Getting formal with dopamine and reward. Neuron 36, 241-263 (2002).
42. Montague, P. R., Dayan, R, Person, C. & Sejnowski, T. Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377, 725-728 (1995).
43. Barto, A. G. in Models of Information Processing in the Basal Ganglia (eds Houk, J. C., Davis, J. L. & Beiser, D. G.) 215-232 (MIT Press, Cambridge, Massachusetts, 1995).
44. Kawagoe, R., Takikawa, Y. & Hikosaka, O. Expectation of reward modulates cognitive signals in the basal ganglia. Nature Neurosci. 1, 411-416 (1998).
45. Lauwereyns, J., Watanabe, K., Coe, B. & Hikosaka, O. A neural correlate of response bias in monkey caudate nucleus. Nature 418, 413-417 (2002).
46. Gold, J. I. Linking reward expectation to behavior in the basal ganglia. Trends Neurosci. 26, 12-14 (2003).
47. Montague, P. R. & Berns, G. S. Neural economics and the biological substrates of valuation. Neuron 36, 265-284 (2002). Among the first of a series of recent attempts (see also references 41,48,49) to integrate our emerging understanding of the role of dopamine with other reward-related neural signals for the purpose of developing a coherent framework for understanding the neural basis of reward and valuation.
48. McClure, S. M., Daw, N. D. & Montague, P. R. A computational substrate for incentive salience. Trends Neurosci. 26, 423-428 (2003).
49. Montague, P. R., Hyman, S. E. & Cohen, J. D. Computational roles for dopamine in behavioral control. Nature 431, 760-767 (2004).
50. Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269-280 (2004).
51. Rolls, E. T. Critchley, H., Mason, R. & Wakeman, E. A. Orbitofrontal cortex neurons: role in olfactory and visual association learning. J. Neurophysiol. 75, 1970-1978 (1996).
52. Tremblay, L. & Schultz, W. Relative reward preference in primate orbitofrontal cortex. Nature 398, 704-708 (1999).
53. Tremblay, L. & Schultz, W. Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 83, 1877-1885 (2000).
54. Watanabe, M. Reward expectancy in primate prefrontal neurons. Nature 382, 629-632 (1996).
55. Leon, M. I. & Shadlen, M. N. Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque. Neuron 24, 415-425 (1999).
56. Kobayashi, S., Lauwereyns, J., Koizumi, M., Sakagami, M. & Hikosaka, O. Influence of reward expectation on visuospatial processing in macaque lateral prefrontal cortex. J. Neurophysiol. 87, 1488-1498 (2002).
57. Wallis, J. D. & Miller, E. K. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur J. Neurosci. 18, 2069-2081 (2003).
58. Roesch, M. R. & Olson, C. R. Impact of expected reward on neuronal activity in prefrontal cortex, frontal and supplementary eye fields and premotor cortex. J. Neurophysiol. 90, 1766-1789 (2003).
59. Roesch, M. R. & Olson, C. R. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304, 307-310 (2004).
60. Shidara, M. & Richmond, B. J. Anterior cingulate: single neuron signals related to the degree of reward expectancy. Science 296, 1709-1711 (2002).
61. McCoy, A. N., Crowley, J. C., Haghighian, G., Dean, H. L. & Platt, M. L. Saccade reward signals in posterior cingulate cortex. Neuron 40, 1031-1040 (2003).
62. Ito, S., Stuphom, V., Brown, J. W. & Schall, J. D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120-122 (2003).
63. Fuster, J. in Cerebral Cortex (eds Peter, J. & Jones, E.) 151-177 (Plenum, New York, 1985).
64. Segraves, M. A. & Goldberg, M. E. Functional properties of corticotectal neurons in the monkey's frontal eye field. J. Neurophysiol. 58, 1387-1419 (1987).
65. Funahashi, S., Bruce, C. J. & Gobman-Rakic, P. S. Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. J. Neurophysiol. 61, 331-349 (1989).
66. Romo, R., Brody, C. D., Hernandez, A. & Lemus, L. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature 339, 470-473 (1999).
67. Gnadt, J. W. & Andersen, R. A. Memory related motor planning activity in posterior parietal cortex of macaque. Exp. Brain Res. 70, 216-220 (1988).
68. Barash, S., Bracewell, R. M., Fogassi, L., Gnadt, J. W. & Andersen, R. A. Saccade-related activity in the lateral intraparietal area. I. Temporal properties; comparison with area 7a. J. Neurophysiol. 66, 1095-1108 (1991).
69. Colby, C. L., Duhamel, J. R. & Goldberg, M. E. Visual, presaccadic, and cognitive activation of single neurons in monkey intraparietal area. J. Neurophysiol. 76, 2841-2852 (1996).
70. Bechara, A., Tranel, D., Damasio, H. & Damasio, A. R. Deciding advantageously before knowing the advantageous strategy. Science 275, 1293-1295 (1997).
71. Iversen, S. D. & Mishkin, M. Perseverative interference in monkeys following selective lesions of the inferior prefrontal convexity. Exp. Brain Res. 11, 376-386 (1970).
72. Rolls, E. T. The orbitofrontal cortex and reward. Cereb. Cortex 10, 284-294 (2000).
73. Damasio, A. R. Descartes' Error: Emotion, Reason, and the Human Brain (Grosset/Putnam, New York, 1994).
74. Musallam, S., Cornel, B. D., Greger, B., Scherberger, H. & Andersen, R. A. Cognitive control signals for neural prosthetics. Science 305, 258-262 (2004).
75. Schultz, W. Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioral ecology. Curr. Opin. Neurobiol. 14, 139-147 (2004). An up-to-date survey of the types and locations of reward-related signals that have been documented using neurophysiological techniques.
76. Barraclough, D. J., Conroy, M. L. & Lee, D. Prefrontal cortex and decision making in a mixed-strategy game. Nature Neurosci. 7, 404-410 (2004).
77. Sugrue, L. P., Corrado, G. S. & Newsome, W. T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782-1787 (2004).
78. Dorris, M. C. & Glimcher, P. W. ActMy in posterior parietal cortex is correlated with the relative subjective desirability of action. Neuron 44, 365-378 (2004). References 76-78 describe studies that all used a novel free-choice approach to investigate the behavioural and neural basis of value-based decision making in awake monkeys. These papers are discussed at length in the main text of this review.
79. Erev, I. & Roth, A. E. Predicting how people play games: reinforcement learning in experimental games with unique, mixed strategy equilibria. Am. Econ. Rev. 88, 848-881 (1998).
80. Platt, M. L. & Glimcher, P. W. Response fields of intraparietal neurons quantified with multiple saccadic targets. Exp. Brain Res. 121, 65-75 (1998).
81. Gottlieb, J. P., Kusunoki, M. & Goldberg, M. E. The representation of visual salience in monkey parietal cortex. Nature 391, 481-484 (1998).
82. Colby, C. L. & Goldberg, M. E. Space and attention in parietal cortex. Annu. Rev. Neurosci. 22, 319-349 (1999).
83. Glimcher, P. W. The neurobiology of visual-saccadic decision making. Annu. Rev. Neurosci. 26, 133-179 (2003).
84. Glimcher, P. W., Dorris, M. C. & Bayer, H. M. Physiological utility theory and the neuroeconomics of choice. Games Ebon. Behav. (in the press).
85. Gold, J. I. & Shadlen, M. N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron 36, 299-308 (2002).
86. Gold, J. I. & Shadlen, M. N. Representation of a perceptual decision in developing oculomotor commands. Nature 404, 390-394 (2000).
87. Shadlen, M. N. Pursuing commitments. Nature Neurosci. 5, 819-821 (2002).
88. Horwitz, G. D., Batista, A. P. & Newsome, W. T. Representation of an abstract perceptual decision in macaque superior colliculus. J. Neurophysiol. 91, 2281-2296 (2004).
89. Rizzolatti, G. & Craighero, L. The mirror-neuron system. Annu. Rev. Neurosci. 27, 169-192 (2004).
90. Maunsell, J. H. Neuronal representations of cognitive state: reward of attention? Trends Cogn. Sci. 8, 261-265 (2004).
91. Koch, C. & Ullman, S. Shifts in selective visual attention: toward the underlying neural circuitry. Hum. Neurobiol. 4, 219-227 (1985).
92. Itti, L. & Koch, C. Computational modeling of visual attention. Nature Rev. Neurosci. 2, 194-203 (2001).
93. Kahneman, D. in Les Prix Nobel 2002 (ed. Frangsmyr, T.) 416-499 (2002). A clear summary of the pioneering work of Kahneman and Tversky on the psychology of valuation and choice, which emphasizes the importance of intuitive over deliberative decision making and challenges basic economic assumptions of rationality.
94. Dayan, R & Balleine, B. W. Reward, motivation, and reinforcement learning. Neuron 36, 285-298 (2002).
95. Dickinson, A. in Animal Learning and Cognition (ed. Mackintosh, N. J.) 45-79 (Academic, Orlando, 1994).
96. Nevin, J. A. Analyzing Thorndike's law of effect: the question of stimulus-response bonds. J. Exp. Anal. Behav. 72, 447-450 (1999).
97. Colwill, R. M. & Rescorla, R. A. Instrumental responding remains sensitive to reinforcer devaluation after extensive training. J. Exp. Psychology Anim. Behav. Process. 11, 520-536 (1985).
98. Nash, J. P. Equilibrium points in n-person games. Proc. Natl Acad. Sci. USA 36, 48-49 (1950).
99. Holt, C. A. & Roth, A. E. The Nash equilibrium: a perspective. Proc. Natl Acad. Sci. USA 101, 3999-4002 (2004).
100. Herrnstein, R. J. Relative and absolute strength of response as a function of frequency of reinforcement. J. Exp. Anal. Behav. 4, 267-272 (1961).
101. Stephens, D. W. & Krebs, J. R. Foraging Theory (Princeton Univ. Press, Princeton, New Jersey, 1986).
102. Baum, W. M. Optimization and the matching law as accounts of instrumental behavior. J. Exp. Anal. Behav. 36, 387-401 (1981).
103. Gallistel, C. R. & Gibbon, J. Time, rate and conditioning. Psychol. Rev. 107, 289-344 (2000). A refreshing, if controversial, look at the subject of operant choice. Argues for distinct decision mechanisms in choice situations that involve opting for the single best alternative, compared with those settings in which an animal must decide in what proportion to allocate its time or responses between alternatives.
104. Heyman, G. M. A Markov model description of changeover probabilities on concurrent variable-interval schedules. J. Exp. Anal. Behav. 31, 41-51 (1979).
105. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, New York, 1966). A seminal work that contributed the key theoretical foundation to research on perceptual decision making.
106. Van Essen, D. C. in The Visual Neurosciences (eds Chalupa, L. & Werner, J. S.) 507-521 (MIT Press, Cambridge, Massachusetts, 2004).
107. Van Essen, D. C. et al. An integrated software suite for surface-based analyses of cerebral cortex. J. Am. Med. Inform. Assoc. 8, 443-459 (2001).