A theoretical account of cognitive effects in delay discounting

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 1052-1064

  Kurth Nelson, Zeb ^a   Bickel, Warren ^b   Redish, A David ^c  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b VIRGINIA TECH CARILION SCHOOL OF MEDICINE   (United States)

^c UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Cognitive resources; Delay discounting; Episodic future thinking; Working memory

Indexed keywords

ARTICLE; ATTENUATION; COGNITION; DECISION MAKING; HUMAN; NEUROPHYSIOLOGY; PRIORITY JOURNAL; REWARD; THEORETICAL STUDY; WORKING MEMORY;

CHOICE BEHAVIOR; COGNITION; DECISION MAKING; FORECASTING; HUMANS; MODELS, PSYCHOLOGICAL; REWARD; TIME FACTORS;

EID: 84859368712     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2012.08058.x     Document Type: Article

References (117)

1. Addis, D.RWong, A.T. & Schacter, D.L. (2007) Remembering the past and imagining the future: common and distinct neural substrates during event construction and elaboration. Neuropsychologia, 45, 1363-1377.
2. Ainslie, G. (1992) Picoeconomics: The Strategic Interaction of Successive Motivational States Within the Person. Cambridge University Press, Cambridge, UK.
3. Ainslie, G. (2001) Breakdown of Will. Cambridge University Press, Cambridge, UK.
4. Anker, J.JPerry, J.LGliddon, L.A. & Carroll, M.E. (2009) Impulsivity predicts the escalation of cocaine self-administration in rats. Pharmacol. Biochem. Behav93, 343-348.
5. Atance, C.M. & O'Neill, D.K. (2001) Episodic future thinking. Trends Cogn. Sci5, 533-539.
6. Baddeley, A.D. (1986) Working Memory. Clarendon Press; Oxford University Press, Oxford.
7. Ballard, KHoude, SSilver-Babaus, S. & McClure, S.M. (2009) The decimal effect: nucleus accumbens activation predicts within-subject increases in temporal discounting rates. Program No. 93.19. 2009 Neuroscience Meeting Planner. Chicago, IL. Society for Neuroscience. Online.
8. Balleine, B.W. & Ostlund, S.B. (2007) Still at the choice-point: action selection and initiation in instrumental conditioning. Ann. N.Y. Acad. Sci1104, 147-171.
9. Barnes, C.A. (1979) Memory deficits associated with senescence: a neurophysiological and behavioral study in the rat. J. Comp. Physiol. Psychol93, 74-104.
10. Barto, A.G. & Mahadevan, S. (2003) Recent advances in hierarchical reinforcement learning. Discrete Event Dyn. Syst13, 341-379.
11. Baumeister, R.FHeatherton, T.F. & Tice, D.M. (1994) Losing Control: How and Why People Fail at Self-Regulation. Academic Press, San Diego.
12. Bechara, ANader, K. & van der Kooy, D. (1998) A two-separate-motivational-systems hypothesis of opioid addiction. Pharmacol. Biochem. Behav59, 1-17.
13. Bechara, ADolan, SDenburg, NHindes, AAnderson, S.W. & Nathan, P.E. (2001) Decision-making deficits, linked to a dysfunctional ventromedial prefrontal cortex, revealed in alcohol and stimulant abusers. Neuropsychologia, 39, 376-389.
14. Benoit, R.GGilbert, S.J. & Burgess, P.W. (2011) A neural mechanism mediating the impact of episodic prospection on farsighted decisions. J. Neurosci31, 6771-6779.
15. Bickel, W.K. & Marsch, L.A. (2001) Toward a behavioral economic understanding of drug dependence: delay discounting processes. Addiction, 96, 73-86.
16. Bickel, W.KYi, RKowal, B.P. & Gatchalian, K.M. (2008) Cigarette smokers discount past and future rewards symmetrically and more than controls: is discounting a measure of impulsivity? Drug Alcohol Depend96, 256-262.
17. Bickel, W.KYi, RLandes, R.DHill, P.F. & Baxter, C. (2011) Remember the future: working memory training decreases delay discounting among stimulant addicts. Biol. Psychiatry, 69, 260-265.
18. Botvinick, M.M. & An, J. (2009). Goal-directed decision making in prefrontal cortex: a computational framework. In Koller, DBengio, Y.YSchuurmans, DBouttou, L. & Culotta, A. (Eds), Advances in Neural Information Processing Systems (NIPS). MIT Press, Cambridge, MA, pp. 169-176.
19. Buckner, R.L. (2010) The role of the hippocampus in prediction and imagination. Annu. Rev. Psychol61, 27.
20. Buckner, R.L. & Carroll, D.C. (2007) Self-projection and the brain. Trends Cogn. Sci11, 49-57.
21. Burks, S.VCarpenter, J.PGoette, L. & Rustichini, A. (2009) Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proc. Natl. Acad. Sci. USA, 106, 7745-7750.
22. Daw, N.D. (2003) Reinforcement Learning Models of the Dopamine System and Their Behavioral Implications. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA.
23. Daw, N.DNiv, Y. & Dayan, P. (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci8, 1704-1711.
24. Daw, N.DGershman, S.JSeymour, BDayan, P. & Dolan, R.J. (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215.
25. Doya, K. (1999) What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw12, 961.
26. Durstewitz, DSeamans, J.K. & Sejnowski, T.J. (2000) Neurocomputational models of working memory. Nat. Neurosci3, 1184-1191.
27. Durstewitz, DVittoz, N.MFloresco, S.B. & Seamans, J.K. (2010) Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning. Neuron, 66, 438-448.
28. Ebert, J.E.J. & Prelec, D. (2007) The fragility of time: time-insensitivity and valuation of the near and far future. Manag. Sci53, 1423-1438.
29. Eichenbaum, HYonelinas, A.P. & Ranganath, C. (2007) The medial temporal lobe and recognition memory. Annu. Rev. Neurosci30, 123-152.
30. Franco-Watkins, A.MPashler, H. & Rickard, T.C. (2006) Does working memory load lead to greater impulsivity? Commentary on Hinson, Jameson, and Whitney (2003) J. Exp. Psychol. Learn. Mem. Cogn32, 443-447.
31. Frank, M.JMoustafa, A.AHaughey, H.MCurran, T. & Hutchison, K.E. (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc. Natl. Acad. Sci. USA, 104, 16311-16316.
32. Frank, M.JDoll, B.BOas-Terpstra, J. & Moreno, F. (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat. Neurosci12, 1062-1068.
33. Gailliot, M.TBaumeister, R.FDeWall, C.NManer, J.KPlant, E.ATice, D.MBrewer, L.E. & Schemeichel, B.J. (2007) Self-control relies on glucose as a limited energy source: willpower is more than a metaphor. J. Pers. Soc. Psychol92, 325-336.
34. Gilbert, D.T. & Wilson, T.D. (2007) Prospection: experiencing the future. Science, 317, 1351-1354.
35. Glimcher, P.W. (2008) Neuroeconomics: Decision Making and the Brain. Academic Press, London.
36. Gold, J.I. & Shadlen, M.N. (2002) Banburismus and the brain: decoding the relationship between sensory stimuli, decisions, and reward. Neuron, 36, 299-308.
37. Green, LMyerson, J. & Calvert, A.L. (2010) Pigeons' discounting of probabilistic and delayed reinforcers. J. Exp. Anal. Behav94, 113-123.
38. Hassabis, DKumaran, DVann, S.D. & Maguire, E.A. (2007) Patients with hippocampal amnesia cannot imagine new experiences. Proc. Natl. Acad. Sci. USA, 104, 1726-1731.
39. Hertz, JKrogh, A. & Palmer, R.G. (1991) Introduction to the Theory of Neural Computation. Addison-Wesley Pub. CoRedwood City, CA.
40. Heyman, G.M. (2009) Addiction: A Disorder of Choice. Harvard University Press, Cambridge, MA.
41. Hill, C. (2008) The rationality of preference construction (and the irrationality of rational choice). Minn. J. Law, SciTech9, 689-742.
42. Hinson, J.MJameson, T.L. & Whitney, P. (2003) Impulsive decision making and working memory. J. Exp. Psychol. Learn. Mem. Cogn29, 298-306.
43. Hopfield, J.J. (1982) Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA, 79, 2554-2558.
44. Howard, M.WFotedar, M.SDatey, A.V. & Hasselmo, M.E. (2005) The temporal context model in spatial navigation and relational learning: toward a common explanation of medial temporal lobe function across domains. Psychol. Rev112, 75-116.
45. Huys, Q.J. (2007) Reinforcers and Control: Towards a Computational Aetiology of Depression. University College London, London.
46. Huys, Q.JEshel, NO'Lions, ESheridan, LDayan, P. & Roiser, J.P. (2012) Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Comput. Biol8, e1002410
47. Johnson, A. & Redish, A.D. (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J. Neurosci27, 12176-12189.
48. Johnson, Avan der Meer, M.A. & Redish, A.D. (2007) Integrating hippocampus and striatum in decision-making. Curr. Opin. Neurobiol17, 692-697.
49. Kaelbling, L.PLittman, M.L. & Moore, A.W. (1996) Reinforcement learning: a survey. J. Artif. Intell. Res4, 237-285.
50. Kahneman, D. & Tversky, A. (1979) Prospect theory: an analysis of decision under risk. Econometrica, 47, 263-292.
51. Kliegel, MMcDaniel, M.A. & Einstein, G.O. (2008) Prospective memory cognitive, neuroscience, developmental, and applied perspectives. Lawrence Erlbaun Associates, New York, NY.
52. Kohonen, T. (1980) Content-Addressable Memories. Springer-Verlag, Berlin; New York.
53. Krajbich, IRangel, A. & Armel, C. (2010) Visual fixations and the computation and comparison of value in simple choice. Nat. Neurosci13, 1292-1298.
54. Kurth-Nelson, Z. & Redish, A.D. (2009) Temporal-difference reinforcement learning with distributed representations. PLoS ONE, 4, e7362.
55. Kwan, DCraver, C.FGreen, LMyerson, JBoyer, P. & Rosenbaum, R.S. (2011) Future decision-making without episodic mental time travel. Hippocampus, in press.
56. Laibson, D. (1997) Golden eggs and hyperbolic discounting. Q. J. Econ112, 443-477.
57. Lauwereyns, J. (2010) The anatomy of bias how neural circuits weigh the options. MIT Press, Cambridge, MA.
58. Loftus, E.F. & Palmer, J.C. (1974) Reconstruction of automobile destruction: an example of the interaction between language and memory. J. Verb. Learn. Verb. Behav13, 585-589.
59. Madden, G.J. & Bickel, W.K. (2010) Impulsivity: The Behavioral and Neurological Science of Discounting. American Psychological Association, Washington, DC.
60. Mazur, J.E. (1997) Choice, delay, probability, and conditioned reinforcement. Anim. Learn. Behav25, 131.
61. McClure, S.MLaibson, D.ILoewenstein, G. & Cohen, J.D. (2004) Separate neural systems value immediate and delayed monetary rewards. Science (New York, N.Y.), 306, 503-507.
62. McDannald, M.ASchoenbaum, GLucantonio, FBurke, K.A. & Niv, Y. (2011) Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci31, 2700-2705.
63. van der Meer, M.A. & Redish, A.D. (2009) Covert expectation-of-reward in rat ventral striatum at decision points. Front. Integr. Neurosci3, 1.
64. van der Meer, M.A. & Redish, A.D. (2010) Expectancies in decision making, reinforcement learning, and ventral striatum. Front. Neurosci4, 6.
65. van der Meer, M.A. & Redish, A.D. (2011) Ventral striatum: a critical look at models of learning and evaluation. Curr. Opin. Neurobiol21, 387-392.
66. van der Meer, M.AKurth-Nelson, Z. & Redish, A.D. (2012) Information processing in decision-making systems. Neuroscientist, in press.
67. Mischel, W. & Grusec, J. (1967) Waiting for rewards and punishments: effects of time and probability on choice. J. Pers. Soc. Psychol5, 24-31.
68. Mischel, W. & Underwood, B. (1974) Instrumental ideation in delay of gratification. Child Dev45, 1083-1088.
69. Montague, P.RDolan, R.JFriston, K.J. & Dayan, P. (2012) Computational psychiatry. Trends Cogn. Sci16, 72-80.
70. Myerson, JGreen, L. & Morris, J. (2011) Modeling the effect of reward amount on probability discounting. J. Exp. Anal. Behav95, 175-187.
71. Newell, A. & Simon, H.A. (1972) Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ.
72. Nilsson, N.J. (1998) Artificial Intelligence: A New Synthesis. Morgan Kaufmann Publishers, San Francisco, CA.
73. Niv, YJoel, D. & Dayan, P. (2006) A normative perspective on motivation. Trends Cogn. Sci10, 375-381.
74. Odum, AMadden, G. & Bickel, W. (2002) Discounting of delayed health gains and losses by current, never- and ex-smokers of cigarettes. Nicotine Tob. Res4, 295-303.
75. Packard, M.G. & McGaugh, J.L. (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol. Learn. Mem65, 65-72.
76. Padoa-Schioppa, C. & Assad, J.A. (2006) Neurons in the orbitofrontal cortex encode economic value. Nature, 441, 223-226.
77. Pennartz, C.MBerke, J.DGraybiel, A.MIto, RLansink, C.Svan der Meer, M.ARedish, A.DSmith, K.S. & Voorn, P. (2009) Corticostriatal interactions during learning, memory processing, and decision making. J. Neurosci29, 12831-12838.
78. Perry, J.LLarson, E.BGerman, J.PMadden, G.J. & Carroll, M.E. (2005) Impulsivity (delay discounting) as a predictor of acquisition of IV cocaine self-administration in female rats. Psychopharmacology, 178, 2-3.
79. Peters, J. & Büchel, C. (2010) Episodic future thinking reduces reward delay discounting through an enhancement of prefrontal-mediotemporal interactions. Neuron, 66, 138-148.
80. Peters, J. & Büchel, C. (2011) The neural mechanisms of inter-temporal decision-making: understanding variability. Trends Cogn. Sci15, 227-239.
81. Rachlin, H. (2004) The Science of Self-Control. Harvard University Press, Cambridge, MA; London.
82. Rachlin, H. (2006) Notes on discounting. J. Exp. Anal. Behav85, 425-435.
83. Radu, P.TYi, RBickel, W.KGross, J.J. & McClure, S.M. (2011) A mechanism for reducing delay discounting by altering temporal attention. J. Exp. Anal. Behav96, 363-385.
84. Ratcliff, R. & McKoon, G. (2008) The diffusion decision model: theory and data for two-choice decision tasks. Neural Comp20, 873-922.
85. Redish, A.D. (1999) Beyond the cognitive map from place cells to episodic memory. MIT Press, Cambridge, MA.
86. Redish, A.D. & Kurth-Nelson, Z. (2010) Neural models of delay discounting. In Madden, G.J. & Bickel, W.K. (Eds), Impulsivity: The Behavioral and Neurological Science of Discounting. American Psychological Association, Washington, DC, pp. 123-158.
87. Redish, A.DJensen, SJohnson, A. & Kurth-Nelson, Z. (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol. Rev114, 784-805.
88. Redish, A.DJensen, S. & Johnson, A. (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav. Brain Sci31, 415-437.
89. Restle, F. (1957) Discrimination of cues in mazes: a resolution of the place-vs.-response question. Psychol. Rev64, 217-228.
90. Ribas-Fernandes, J.J.FSolway, ADiuk, CMcGuire, J.TBarto, A.GNiv, Y. & Botvinick, M.M. (2011) A neural signature of hierarchical reinforcement learning. Neuron, 71, 370-379.
91. Rich, E. & Knight, K. (1991) Artificial Intelligence. McGraw-Hill, New York.
92. Roesch, M.RSchoenbaum, GSingh, TLeon, B.P. & Mullins, S.E. (2009) Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J. Neurosci29, 13365-13376.
93. Samuelson, P.A. (1937) A note on measurement of utility. Rev. Econ. Stud4, 155-161.
94. Schacter, D.L. & Addis, D.R. (2007) Constructive memory: the ghosts of past and future. Nature, 445, 27.
95. Schacter, D.LAddis, D.R. & Buckner, R.L. (2007) Remembering the past to imagine the future: the prospective brain. Nat. Rev. Neurosci8, 657-661.
96. Schacter, D.LAddis, D.R. & Buckner, R.L. (2008) Episodic simulation of future events: concepts, data, and applications. Ann. N. Y. Acad. Sci1124, 39-60.
97. Schoenbaum, GStalnaker, T.ATakahashi, Y.K. & Roesch, M.R. (2009) A new perspective on the role of the orbitofrontal cortex in adaptive behaviour. Nat. Rev. Neurosci10, 885-892.
98. Schweighofer, NShishida, KHan, C.EOkamoto, YTanaka, S.CYamawaki, S. & Doya, K. (2006) Humans can adopt optimal discounting strategy under real-time constraints. PLoS Comput. Biol2, e152.
99. Seamans, J.K. & Yang, C.R. (2004) The principal features and mechanisms of dopamine modulation in the prefrontal cortex. Prog. Neurobiol74, 1-58.
100. Simon, D.A. & Daw, N.D. (2011) Neural correlates of forward planning in a spatial decision task in humans. J. Neurosci31, 5526-5539.
101. Simonson, I. & Tversky, A. (1992) Choice in context: tradeoff contrast and extremeness aversion. J. Mark. Res29, 281-295.
102. Smith, ALi, MBecker, S. & Kapur, S. (2006) Dopamine, prediction error and associative learning: a model-based account. Network, 17, 61-84.
103. Sozou, P.D. (1998) On hyperbolic discounting and uncertain hazard rates. Proc. Roy. Soc. London B Bio. Sci265, 2015-2050.
104. Stanger, CRyan, S.RFu, HLandes, R.DJones, B.ABickel, W.K. & Budney, A.J. (2011) Delay discounting predicts adolescent substance abuse treatment outcome. Exp. Clin. Psychopharmacolin press.
105. Stephens, D.W. & Krebs, J.R. (1986) Foraging Theory. Princeton University Press, Princeton, NJ.
106. Suri, R.E. (2002) TD models of reward predictive responses in dopamine neurons. Neural Netw15, 523-533.
107. Sutton, R.S. & Barto, A.G. (1998) Reinforcement learning an introduction. MIT Press, Cambridge, MA.
108. Tolman, E.C. (1939) Prediction of vicarious trial and error by means of the schematic sowbug. Psychol. Rev46, 318-336.
109. Trope, Y. & Liberman, N. (2003) Temporal construal. Psychol. Rev110, 403-421.
110. Vohs, K.D. & Faber, R.J. (2007) Spent resources: self-regulatory resource availability affects impulse buying. J. Consum. Res33, 537-547.
111. Wilson, H.R. & Cowan, J.D. (1972) Excitatory and inhibitory interactions in localized populations of model neurons. Biophys. J12, 1-24.
112. Winstanley, CRobbins, T.WBalleine, B.WBrown, J.WBüchel, CCools, RDurstewutz, DRedish, A.DSeamans, J.K. & Robbins, T.W. (2012). Cognitive control, cognitive search, and motivational salience: a systems neuroscience approach. In Todd, P.MHills, T.T. & Robbins, T.W. (Eds), Cognitive Search: Evolution, Algorithms, and the Brain (Strüngmann Forum Report). MIT Press, Cambridge, MA, in press.
113. Yamashita, K. & Tanaka, S. (2005) Parametric study of dopaminergic neuromodulatory effects in a reduced model of the prefrontal cortex. Neurocomputing, 65, 579.
114. Yi, RGatchalian, K.M. & Bickel, W.K. (2006) Discounting of past outcomes. Exp. Clin. Psychopharmacol14, 311-317.
115. Yin, H.H. & Knowlton, B.J. (2004) Contributions of striatal subregions to place and response learning. Learn. Mem11, 459-463.
116. Zald, D.H. & Rauch, S.L. (2006) The Orbitofrontal Cortex. Oxford University Press, Oxford; New York.
117. Zhang, K. (1996) Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. J. Neurosci16, 2112-2126.