Subjective neuronal coding of reward: Temporal value discounting and risk

European Journal of Neuroscience

Volumn 31, Issue 12, 2010, Pages 2124-2135

  Schultz, Wolfram ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Dopamine; Frontal cortex; Human; Monkey; Striatum

Indexed keywords

BRAIN NERVE CELL; CORPUS STRIATUM; DECISION MAKING; DOPAMINERGIC NERVE CELL; FRONTAL CORTEX; HUMAN; INFORMATION PROCESSING; NEUROIMAGING; NEUROPHYSIOLOGY; NONHUMAN; PREFRONTAL CORTEX; PRIORITY JOURNAL; REVIEW; REWARD; TASK PERFORMANCE; VISUAL STIMULATION;

ACTION POTENTIALS; ANIMALS; BEHAVIOR; BRAIN; BRAIN MAPPING; CHOICE BEHAVIOR; DOPAMINE; HUMANS; NEURONS; REWARD; TIME FACTORS;

EID: 77953760906     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2010.07282.x     Document Type: Review

References (65)

1. Ainslie, G.W. (1974) Impulse control in pigeons. J. Exp. Anal. Behav., 21, 485 489.
2. Ainslie, G.W. (1975) Specious rewards: a behavioral theory of impulsiveness and impulse control. Psychol. Bull., 82, 463 496.
3. Boysen, S.T., Berntson, G.G. Mukobi, K.L. (2001) Size matters: impact of item size and quantity on array choice by chimpanzees (Pan troglodytes). J. Comp. Psychol., 115, 106 110.
4. Caraco, T., Martindale, S. Whitham, T.S. (1980) An empirical demonstration of risk-sensitive foraging preferences. Anim. Behav., 28, 820 830.
5. Caraco, T., Blankenhorn, W.U., Gregory, G.M., Newman, J.A., Recer, G.M. Zwicker, S.M. (1990) Risk-sensitivity: ambient temperaure effects foraging choice. Anim. Behav., 39, 338 345.
6. Cardinal, R.N., Pennicott, D.R., Sugathapala, C.L., Robbins, T.W. Everitt, B.J. (2001) Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science, 292, 2499 2501.
7. Critchley, H.G. Rolls, E.T. (1996) Hunger and satiety modify the responses of olfactory and visual neurons in the primate orbitofrontal cortex. J. Neurophysiol., 75, 1673 1686.
8. Fecteau, S., Pascual-Leone, A., Zald, D.H., Liguori, P., Théoret, H., Boggio, P.S. Fregni, F. (2007) Activation of prefrontal cortex by transcranial direct current stimulation reduces appetite for risk during ambiguous decision making. J. Neurosci., 27, 6212 6218.
9. Fiorillo, C.D., Tobler, P.N. Schultz, W. (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898 1902.
10. Fiorillo, C.D., Newsome, W.T. Schultz, W. (2008) The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci., 11, 966 973.
11. Gregorios-Pippas, L., Tobler, P.N. Schultz, W. (2009) Short term temporal discounting of reward value in human ventral striatum. J. Neurophysiol., 101, 1507 1523.
12. Hariri, A.R., Brown, S.M., Williamson, D.E., Flory, J.D., de Wit, H. Manuck, S.B. (2006) Preference for immediate over delayed rewards is associated with magnitude of ventral striatal activity. J. Neurosci., 26, 13213 13217.
13. Ho, M.-Y., Mobini, S., Chinang, T.-J., Bradshaw, C.M. Szabadi, E. (1999) Theory and method in the quantitative analysis of 'impulsive choice' behaviour: implications for psychopharmacology. Psychopharmacology, 146, 362 372.
14. Holland, P.C. (1980) CS-US interval as a determinant of the form of Pavlovian appetitive conditioned responses. J. Exp. Psychol. Anim. Behav. Process., 6, 155 174.
15. Hollerman, J.R., Tremblay, L. Schultz, W. (1998) Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol., 80, 947 963. (Pubitemid 28387672)
16. Johnson, M.W. Bickel, W.K. (2002) Within-subject comparison of real and hypothetical money rewards in delay discounting. J. Exp. Anal. Behav., 77, 129 146.
17. Kable, J.W. Glimcher, P.W. (2007) The neural correlates of subjective value during intertemporal choice. Nat. Neurosci., 10, 1625 1633.
18. Kheramin, S., Body, S., Ho, M.Y., Velázquez-Martinez, D.N., Bradshaw, C.M., Szabadi, E., Deakin, J.F.W. Anderson, I.M. (2004) Effects of orbital prefrontal cortex dopamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology, 175, 206 214.
19. Kim, S., Hwang, J., Seo, H. Lee, D. (2009) Valuation of uncertain and delayed rewards in primate prefrontal cortex. Neural Netw., 22, 294 304.
20. Kirby, K.N. Marakovic, N.N. (1995) Modeling myopic decisions: evidence for hyperbolic delay-discounting within subjects and amounts. Organ. Behav. Hum. Decis. Process., 64, 22 30.
21. Knoch, D., Gianotti, L.R.R., Pascual-Leone, A., Treyer, V., Regard, M., Hohmann, M. Brugger, P. (2006) Disruption of right prefrontal cortex by low-frequency repetitive transcranial magnetic stimulation induces risk-taking behavior. J. Neurosci., 26, 6469 6472.
22. Knutson, B., Taylor, J., Kaufman, M., Peterson, R. Glover, G. (2005) Distributed neural representation of expected value. J. Neurosci., 25, 4806 4812.
23. Kobayashi, S. Schultz, W. (2008) Influence of reward delays on responses of dopamine neurons. J. Neurosci., 28, 7837 7846.
24. Kreps, D.M. (1990) A Course in Microeconomic Theory. Pearson Education Ltd
25. Levy, H. Markowitz, H.M. (1979) Approximating expected utility by a function of mean and variance. Am. Econ. Rev., 69, 308 317.
26. Loewenstein, G. Prelec, D. (1992) Anomalies in intertemporal choice: evidence and an interpretation. Q. J. Econ., 107, 573 597.
27. Luce, R.D. (2000) Utility of Gains and Losses: Measurement-Theoretical and Experimental Approaches. Lawrence Erlbaum
28. Markowitz, H. (1952) Portfolio selection. J. Finance, 7, 77 91.
29. McClure, S.M., Laibson, D.I., Loewenstein, G. Cohen, J.D. (2004) Separate neural systems value immediate and delayed monetary rewards. Science, 306, 503 507.
30. McClure, S.M., Ericson, K.M., Laibson, D.I., Loewenstein, G. Cohen, J.D. (2007) Time discounting for primary rewards. J. Neurosci., 27, 5796 5804.
31. Meck, W.H. (2005) Neuropsychology of timing and time perception. Brain Cogn., 58, 1 8.
32. O'Doherty, J. (2004) Reward representations and reward-related learning in the human brain: insights from neuroimaging. Curr. Opin. Neurobiol., 14, 769 776.
33. Pavlov, P.I. (1927) Conditioned Reflexes. Oxford University Press
34. Preuschoff, K., Bossaerts, P. Quartz, S.R. (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron, 51, 381 390.
35. Raby, C.R., Alexis, D.M., Dickinson, A. Clayton, N.S. (2007) Planning for the future by western scrub-jays. Nature, 445, 919 921.
36. Rescorla, R.A. (1967) Pavlovian conditioning and its proper control procedures. Psychol. Rev., 74, 71 80.
37. Richards, J.B., Mitchell, S.H., de Wit, H. Seiden, L.S. (1997) Determination of discount functions in rats with an adjusting-amount procedure. J. Exp. Anal. Behav., 67, 353 366.
38. Roberts, S. (1981) Isolation of an internal clock. J. Exp. Psychol. Anim. Behav. Process., 7, 242 268.
39. Rodriguez, M.L. Logue, A.W. (1988) Adjusting delay to reinforcement: comparing choice in pigeons and humans. J. Exp. Psychol. Anim. Behav. Process., 14, 105 117.
40. Roesch, M.R. Olson, C.R. (2005a) Neuronal activity dependent on anticipated and elapsed delay in macaque prefrontal cortex, frontal and supplementary eye fields, and premotor cortex. J. Neurophysiol., 94, 1469 1497.
41. Roesch, M.R. Olson, C.R. (2005b) Neuronal activity in orbitofrontal cortex reflects the value of time. J. Neurophysiol., 94, 2457 2471.
42. Roesch, M.R., Taylor, A.R. Schoenbaum, G. (2006) Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron, 51, 509 520.
43. Roesch, M.R., Calu, D.J. Schoenbaum, G. (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci., 10, 1615 1624.
44. Roesch, M.R., Singh, T., Brown, P.L., Mullins, S.E. Schoenbaum, G. (2009) Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J. Neurosci., 29, 13365 13376.
45. Samejima, K., Ueda, Y., Doya, K. Kimura, M. (2005) Representation of action-specific reward values in the striatum. Science, 310, 1337 1340.
46. Samuelson, P.A. (1937) Some aspects of the pure theory of capital. Q. J. Econ., 51, 469 496.
47. Schultz, W. Romo, R. (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J. Neurophysiol., 63, 607 624. (Pubitemid 20115487)
48. Sheafor, P.J. (1975) Pseudoconditioned jaw movements of the rabbit reflect associations conditioned to contextual background cues. J. Exp. Psychol. Anim. Behav. Process., 104, 245 260.
49. Sutton, R.S. Barto, A.G. (1998) Reinforcement Learning. MIT Press
50. Tanaka, S.C., Doya, K., Okada, G., Ueda, K., Okamoto, Y. Yamagaki, S. (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci., 7, 887 893.
51. Tobler, P.N., Dickinson, A. Schultz, W. (2003) Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci., 23, 10402 10410.
52. Tobler, P.N., Fiorillo, C.D. Schultz, W. (2005) Adaptive coding of reward value by dopamine neurons. Science, 307, 1642 1645.
53. Tobler, P.N., Fletcher, P.C., Bullmore, E.T. Schultz, W. (2007a) Learning-related human brain activations reflecting individual finances. Neuron, 54, 167 175.
54. Tobler, P.N., O'Doherty, J.P., Dolan, R. Schultz, W. (2007b) Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol., 97, 1621 1632.
55. Tobler, P.N., Christopoulos, G.I., O'Doherty, J.P., Dolan, R.J. Schultz, W. (2009) Risk-dependent reward value signal in human prefrontal cortex. Proc. Natl Acad. Sci. USA, 106, 7185 7190.
56. Waelti, P., Dickinson, A. Schultz, W. (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43 48.
57. Wakker, P. (1994) Separating marginal utility and probabilistic risk aversion. Theor. Decis., 36, 1 44.
58. Watanabe, M. (1996) Reward expectancy in primate prefrontal neurons. Nature, 382, 629 632.
59. Watanabe, M., Cromwell, H.C., Tremblay, L., Hollerman, J.R. Schultz, W. (2001) Behavioral reactions reflecting differential expectations of outcomes in monkeys. Exp. Brain Res., 140, 511 518.
60. Weber, E.U. Milliman, R.A. (1997) Perceived risk attitudes: relating risk perception to risky choice. Manage. Sci., 43, 123 144.
61. Weber, E.U., Blais, A.-R. Betz, N. (2002) A domain-specific risk-attitude scale: measuring risk perceptions and risk behaviors. J. Behav. Decis. Mak., 15, 263 290.
62. Weber, E.U., Shafir, S. Blais, A.-R. (2004) Predicting risk sensitivity in humans and lower animals: risk as variance or coefficient of variation. Psychol. Rev., 111, 430 445.
63. Winstanley, C.A., Theobald, D.E.H., Cardinal, R.N. Robbins, T.W. (2004) Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J. Neurosci., 24, 4718 4722.
64. Wise, R.A. (2002) Brain reward circuitry: insights from unsensed incentives. Neuron, 36, 229 240.
65. Wittmann, M., Leland, D.S. Paulus, M.P. (2007) Time and decision making: differential contribution of the posterior insular cortex and the striatum during a delay discounting task. Exp. Brain Res., 179, 643 653.