Modeling decision-making systems in addiction

Computational Neuroscience of Drug Addiction

Volumn , Issue , 2012, Pages 163-187

  Kurth Nelson, Zeb ^a   Redish, A David ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84870883126     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-1-4614-0751-5_6     Document Type: Chapter

References (161)

1. Ainslie G (1974) Impulse control in pigeons. J Exp Anal Behav 21:485
2. Ainslie G (2001) Breakdown of will. Cambridge University Press, Cambridge
3. Andersen S, Harrison GW, Lau MI, Rutström EE (2008) Eliciting risk and time preferences. Econometrica 76:583
4. Aragona BJ, Cleaveland NA, Stuber GD, Day JJ, Carelli RM, Wightman RM (2008) Preferential enhancement of dopamine transmission within the nucleus accumbens shell by cocaine is attributable to a direct increase in phasic dopamine release events. J Neurosci 28:8821
5. Arbisi PA, Billington CJ, Levine AS (1999) The effect of naltrexone on taste detection and recognition threshold. Appetite 32:241
6. Balleine BW (2001) Incentive processes in instrumental conditioning. In: Handbook of contemporary Learning Theories, p 307
7. Balleine BW (2004) Incentive behavior. In: The behavior of the laboratory rat: a handbook with tests, p 436
8. Balleine BW, Dickinson A (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology 37:407
9. Balleine BW, Daw ND, O'Doherty JP (2008) Multiple forms of value learning and the function of dopamine. In: Neuroeconomics: decision making and the brain, p 367
10. Barnes CA (1979) Memory deficits associated with senscence: A neurophysiological and behavioral study in the rat. J Comp Physiol Psychol 93:74
11. Barto AG (1994) Adaptive critics and the basal ganglia. In: Models of information processing in the basal ganglia, p 215
12. Baum W, Rachlin H (1969) Choice as time allocation. J Exp Anal Behav 12:861
13. Bayer HM, Glimcher P (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129
14. Becker GS, Murphy KM (1988) A theory of rational addiction. J Polit Econ 96:675
15. Becker GS, Grossman M, Murphy KM (1994) An empirical analysis of cigarette addiction. Am Econ Rev 84:396
16. Bernheim BD, Rangel A (2004) Addiction and cue-triggered decision processes. Am Econ Rev 94:1558
17. Berridge KC (2007) The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 191:391
18. Berridge CW, Arnsten AF, Foote SL (1993) Noradrenergic modulation of cognitive function: clinical implications of anatomical, electrophysiological and behavioural studies in animal models. Psychol Med 23:557
19. Bickel WK, Odum AL, Madden GJ (1999) Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers. Psychopharmacology (Berlin) 146:447
20. Bouton ME (2002) Context, ambiguity, and unlearning: sources of relapse after behavioral extinction. Biol Psychiatry 52:976
21. Bouton ME, Swartzentruber D (1989) Slow reacquisition following extinction: context, encoding, and retrieval mechanisms. J Exp Psychol, Anim Behav Processes 15:43
22. Bouton ME, Westbrook RF, Corcoran KA, Maren S (2006) Contextual and temporal modulation of extinction: behavioral and biological mechanisms. Biol Psychiatry 60:352
23. Breland K, Breland M (1961) The misbehavior of organisms. Am Psychol 16:682
24. Burks SV, Carpenter JP, Goette L, Rustichini A (2009) Cognitive skills affect economic preferences, strategic behavior, and job attachment. Proc Natl Acad Sci 106:7745
25. Childress AR, Ehrman R, Rohsenow DJ, Robbins SJ, O'Brien CP (1992) Classically conditioned factors in drug dependence. In: Substance abuse: a comprehensive textbook, p 56
26. Christensen CJ, Silberberg A, Hursh SR, Roma PG, Riley AL (2008) Demand for cocaine and food over time. Pharmacol Biochem Behav 91:209
27. Chung SH, Herrnstein RJ (1967) Choice and delay of reinforcement. J Exp Anal Behav 10:67
28. Corbit LH, Balleine BW (2000) The role of the hippocampus in instrumental conditioning. J Neurosci 20:4233
29. Cote D, Caron A, Aubert J, Desrochers V, Ladouceur R (2003) Near wins prolong gambling on a video lottery terminal. J Gambl Stud 19:433
30. Courville AC (2006) A latent cause theory of classical conditioning. Doctoral dissertation, Carnegie Mellon University
31. Custer RL (1984) Profile of the pathological gambler. J Clin Psychiatry 45:35
32. Daw ND (2003) Reinforcement learning models of the dopamine system and their behavioral implications. Doctoral dissertation, Carnegie Mellon University
33. Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16:199
34. Daw ND, Kakade S, Dayan P (2002) Opponent interactions between serotonin and dopamine. Neural Netw 15:603
35. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704
36. Daw ND, Courville AC, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Comput 18:1637
37. Dayan P (2002) Motivated reinforcement learning. Advances in neural information processing systems: proceedings of the 2002 conference
38. Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285
39. Dayan P, Seymour B (2008) Values and actions in aversion. In: Neuroeconomics: decision making and the brain, p 175
40. Dayan P, Niv Y, Seymour B, Daw ND (2006) The misbehavior of value and the discipline of the will. Neural Netw 19:1153
41. Dezfouli A, Piray P, Keramati MM, Ekhtiari H, Lucas C, Mokri A (2009) A neurocomputational model for cocaine addiction. Neural Comput 21:2869
42. di Chiara G (1999) Drug addiction as dopamine-dependent associative learning disorder. Eur J Pharmacol 375:13
43. Dickerson M, O'Connor J (2006) Gambling as an addictive behavior. Cambridge University Press, Cambridge
44. Domjan M (1998) The principles of learning and behavior. Brooks/Cole
45. Doya K (2000) Metalearning, neuromodulation, and emotion. In: Affective minds, p 101
46. Elster J (1999) Gambling and addiction. In: Getting hooked: rationality and addiction, p 208
47. Estes WK (1943) Discriminative conditioning. I. A discriminative property of conditioned anticipation. J Exp Psychol 32:150
48. Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11:966
49. Flagel SB, Watson SJ, Akil H, Robinson TE (2008) Individual differences in the attribution of incentive salience to a reward-related cue: Influence on cocaine sensitization. Behav Brain Res 186:48
50. Frederick S, Loewenstein G, O'Donoghue T (2002) Time Discounting and time preference: A critical review. J Econ Lit 40:351
51. Fuhs MC, Touretzky DS (2007) Context learning in the rodent hippocampus. Neural Comput 19:3172
52. Gershman SJ, Blei DM, Niv Y (2010) Context, learning, and extinction. Psychol Rev 117:197
53. Glimcher PW, Camerer C, Fehr E, Poldrack RA (2008) Neuroeconomics: decision making and the brain. Elsevier/Academic Press, London
54. Goldman MS, Brown SA, Christiansen BA (1987) Expectancy theory: thinking about drinking. In: Psychological theories of drinking and alcoholism, p 181
55. Goldstein A (2000) Addiction: from biology to drug policy. Oxford University Press, Oxford
56. Grossman M, Chaloupka FJ (1998) The demand for cocaine by young adults: a rational addiction approach. J Health Econ 17:427
57. Gul F, Pesendorfer W (2001) Temptation and self-control. Econometrica 69:1403
58. Gutkin BS, Dehaene S, Changeux JP (2006) A neurocomputational hypothesis for nicotine addiction. Proc Natl Acad Sci USA 103:1106
59. Henly SE, Ostdiek A, Blackwell E, Knutie S, Dunlap AS, Stephens DW (2008) The discountingby-interruptions hypothesis: model and experiment. Behav Ecol 19:154
60. Hershberger WA (1986) An approach through the looking-glass. Anim Learn Behav 14:443
61. Heyman GM (2009) Addiction: a disorder of choice. Harvard University Press, Cambridge
62. Higgins ST, Heil SH, Lussier JP (2004) Clinical implications of reinforcement as a determinant of substance use disorders. Annu Rev Psychol 55:431
63. Hirsh R (1974) The hippocampus and contextual retrieval of information from memory: A theory. Behav Biol 12:421
64. Hirsh R, Leber B, Gillman K (1978) Fornix fibers and motivational states as controllers of behavior: A study stimulated by the contextual retrieval theory. Behav Biol 22:463
65. Hu D, Amsel A (1995) A Simple Test of the Vicarious Trial- and-Error Hypothesis of Hippocampal Function. Proc Natl Acad Sci USA 92:5506
66. Hu D, Xu X, Gonzalez-Lima F (2006) Vicarious trial- and-error behavior and hippocampal cytochrome oxidase activity during Y-maze discrimination learning in the rat. Int J Neurosci 116:265
67. Hunt WA (1998) Pharmacology of alcohol. In: Tarter RE, Ammerman RT, Ott PJ (eds) Handbook of substance abuse: Neurobehavioral pharmacology. Plenum, New York, pp 7-22
68. Isaacson RL (1974) The limbic system. Plenum, New York
69. Isoda M, Hikosaka O (2008) Role for subthalamic nucleus neurons in switching from automatic to controlled eye movement. J Neurosci 28:7209
70. Jaffe JH, Cascella NG, Kumor KM, Sherer MA (1989) Cocaine-induced cocaine craving. Psychopharmacology (Berlin) 97:59
71. Jaffe A, Gitisetan S, Tarash I, Pham AZ, Jentsch JD (2010) Are nicotine-related cues susceptible to the blocking effect? Society for Neuroscience Abstracts, Program Number 268.4
72. Johnson A, Redish AD (2007) Neural ensembles in CA3 transiently encode paths forward of the animal at a decision point. J Neurosci 27:12176
73. Jones BT, Corbin W, Fromme K (2001) A review of expectancy theory and alcohol consumption. Addiction 96:57
74. Kamin LJ (1969) Predictability, surprise, attention, and conditioning. In: Learning in animals, p 279
75. Kirby KN, Herrnstein RJ (1995) Preference reversals due to myopic discounting of delayed reward. Psychol Sci 6:83
76. Kruse JM, Overmier JB, Konz WA, Rokke E (1983) Pavlovian conditioned stimulus effects upon instrumental choice behavior are reinforcer specific. Learn Motiv 14:165
77. Kuhar MJ, Ritz MC, Sharkey J (1988) Cocaine receptors on dopamine transporters mediate cocaine-reinforced behavior. In: Mechanisms of cocaine abuse and toxicity, p 14
78. Kurth-Nelson Z, Redish AD (2009) Temporal-difference reinforcement learning with distributed representations. PLoS ONE 4:e7362
79. Kurth-Nelson Z, Redish AD (2010) A reinforcement learning model of precommitment in decision making. Frontiers Behav Neurosci 4:184
80. Langer EJ, Roth J (1975) Heads I win, tails it's chance: The illusion of control as a function of the sequence of outcomes in a purely chance task. J Pers Soc Psychol 32:951
81. Lebron K, Milad MR, Quirk GJ (2004) Delayed recall of fear extinction in rats with lesions of ventral medial prefrontal cortex. Learn Mem 11:544
82. Lenoir M, Serre F, Cantin L, Ahmed SH (2007) Intense sweetness surpasses cocaine reward. PLoS ONE 2:e698
83. Levine AS, Billington CJ (2004) Opioids as agents of reward-related feeding: a consideration of the evidence. Physiol Behav 82:57
84. Liao D, Lin H, Law PY, Loh HH (2005) Mu-opioid receptors modulate the stability of dendritic spines. Proc Natl Acad Sci USA 102:1725
85. Liu J, Liu J, Hammit JK, Chou S. (1999) The price elasticity of opium in Taiwan, 1914-1942. J Health Econ 18:795
86. Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145
87. Lovibond PF (1983) Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. J Exp Psychol Anim Behav Process 9:225
88. Mackintosh NJ (1974) The psychology of animal learning. Academic Press, San Diego
89. Madden GJ, Bickel WK (2010) Impulsivity: the behavioral and neurological science of discounting. American Psychological Association, Washington, DC
90. Mazur J (1987) An adjusting procedure for studying delayed reinforcement. In: Quantitative analyses of behavior, p 55
91. McCaul ME, Petry NM (2003) The role of psychosocial treatments in pharmacotherapy for alcoholism. Am J Addict 12:S41
92. McFarland K, Kalivas PW (2001) The circuitry mediating cocaine-induced reinstatement of drugseeking behavior. J Neurosci 21:8655
93. Milad MR, Vidal-Gonzalez I, Quirk GJ (2004) Electrical stimulation of medial prefrontal cortex reduces conditioned fear in a temporally specific manner. Behav Neurosci 118:389
94. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936
95. Moos RH, Moos BS (2004) Long-term influence of duration and frequency of participation in alcoholics anonymous on individuals with alcohol use disorders. J Consult Clin Psychol 72:81
96. Moos RH, Moos BS (2006a) Participation in treatment and Alcoholics Anonymous: a 16-year follow-up of initially untreated individuals. J Clin Psychol 62:735
97. Moos RH, Moos BS (2006b) Rates and predictors of relapse after natural and treated remission from alcohol use disorders. Addiction 101:212
98. Muenzinger KF (1938) Vicarious trial and error at a point of choice. I. A general survey of its relation to learning efficiency. J Genet Psychol 53:75
99. Nadel L, Willner J (1980) Context and conditioning: A place for space. Physiol Psychol 8:218
100. Nestler EJ (1996) Under siege: The brain on opiates. Neuron 16:897
101. Niv Y, Montague PR (2008) Theoretical and empirical studies of learning. In: Neuroeconomics: decision making and the brain, p 331
102. Niv Y, Daw ND, Dayan P (2006) Choice values. Nat Neurosci 9:987
103. O'Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452
104. O'Keefe J, Dostrovsky J (1971) The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely moving rat. Brain Res 34:171
105. O'Keefe J, Nadel L (1978) The hippocampus as a cognitive map. Clarendon, Oxford
106. Oscar-Berman M, Marinkovic K (2003) Alcoholism and the brain: an overview. Alcohol Res Health 27(2):125-134
107. Ostlund SB, Balleine BW (2008) The disunity of Pavlovian and instrumental values. Behav Brain Sci 31:456
108. Packard MG, McGaugh JL (1996) Inactivation of hippocampus or caudate nucleus with lidocaine differentially affects expression of place and response learning. Neurobiol Learn Mem 65:65
109. Paine TA, Dringenberg HC, Olmstead MC (2003) Effects of chronic cocaine on impulsivity: relation to cortical serotonin mechanisms. Behav Brain Res 147:135
110. Panlilio LV, Thorndike EB, Schindler CW (2007) Blocking of conditioning to a cocaine-paired stimulus: Testing the hypothesis that cocaine perpetually produces a signal of larger-thanexpected reward. Pharmacol Biochem Behav 86:774
111. Parke J, Griffiths M (2004) Gambling addiction and the evolution of the near miss. Addict Res Theory 12:407
112. Pavlov I (1927) Conditioned reflexes. Oxford Univ Press, Oxford
113. Phillips PEM, Stuber GD, Heien MLAV, Wightman RM, Carelli RM (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614
114. Porrino LJ, Lyons D, Smith HR, Daunais JB, Nader MA (2004) Cocaine self-administration produces a progressive involvement of limbic, association, and sensorimotor striatal domains. J Neurosci 24:3554
115. Preuschoff K, Bossaerts P, Quartz SR (2006) Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51:381
116. Quirk GJ, Garcia R, González-Lima F (2006) Prefrontal mechanisms in extinction of conditioned fear. Biol Psychiatry 60:337
117. Rachlin H (2000) The science of self-control. Harvard University Press, Cambridge
118. Rachlin H, Green L (1972) Commitment, choice, and self-control. J Exp Anal Behav 17:15
119. Redish AD (1999) Beyond the cognitive map: from place cells to episodic memory. MIT Press, Cambridge
120. Redish AD (2004) Addiction as a computational process gone awry. Science 306:1944
121. Redish AD (2009) Implications of the multiple-vulnerabilities theory of addiction for craving and relapse. Addiction 104:1940
122. Redish AD, Johnson A (2007) A computational model of craving and obsession. Ann NY Acad Sci 1104:324
123. Redish AD, Kurth-Nelson Z (2010) Neural models of temporal discounting. In: Impulsivity: the behavioral and neurological science of discounting, p 123
124. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 114:784
125. Redish AD, Jensen S, Johnson A (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 31:415
126. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Classical conditioning II, p 64
127. Restle F (1957) Discrimination of cues in mazes: A resolution of the 'place-vs-response' question. Psychol Rev 64:217
128. Reynolds B, Ortengren A, Richards JB, de Wit H (2006) Dimensions of impulsive behavior: personality and behavioral measures. Pers Individ Differ 40:305
129. Ritz MC, Lamb RJ, Goldberg SR, Kuhar MJ (1987) Cocaine receptors on dopamine transporters are related to self-administration of cocaine. Science 237:1219
130. Robinson TE, Berridge KC (1993) The neural basis of drug craving: An incentive-sensitization theory of addiction. Brains Res Rev 18:247
131. Robinson TE, Berridge KC (2001) Mechanisms of action of addictive stimuli: Incentivesensitization and addiction. Addiction 96:103
132. Robinson TE, Berridge KC (2003) Addiction. Annu Rev Psychol 54:25
133. Robinson TE, Berridge KC (2004) Incentive-sensitization and drug 'wanting'. Psychopharmacology 171:352
134. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36:241
135. Schultz W, Dayan P, Montague R (1997) A neural substrate of prediction and reward. Science 275:1593
136. Schweighofer N, Shishida K, Han CE, Yamawaki S, Doya K (2006) Humans can adopt optimal discounting strategy under real-time constraints. PLoS Comput Biol 2:e152
137. Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards. Theory, experiments, and possible neural mechanisms. Ann NY Acad Sci 1104:289
138. Si J, Barto AG, Powell WB, Wunsch D (2004) Handbook of learning and approximate dynamic programming. Wiley/IEEE Press, New York
139. Simon NW, Mendez IA, Setlow B (2007) Cocaine exposure causes long-term increases in impulsive choice. Behav Neurosci 121:543
140. Smith A, Li M, Becker S, Kapur S (2006) Dopamine, prediction error and associative learning: a model-based account. Network: Comput Neural Syst 17:61
141. Sotres-Bayon F, Cain CK, LeDoux JE (2006) Brain mechanisms of fear extinction: historical perspectives on the contribution of prefrontal cortex. Biol Psychiatry 60:329
142. Sozou PD (1998) On hyperbolic discounting and uncertain hazard rates. R Soc Lond B 265:2015
143. Stahl SM, Pradko JF, Haight BR, Modell JG, Rockett CB, Learned-Coughlin S (2004) A review of the neuropharmacology of bupropion, a dual norepinephrine and dopamine reuptake inhibitor. Prim Care Companion J Clin Psychiat 6:159
144. Strotz RH (1956) Myopia and inconsistency in dynamic utility maximization. Rev Econ Stud 23:165
145. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
146. Talmi D, Seymour B, Dayan P, Dolan RJ (2008) Human Pavlovian instrumental transfer. J Neurosci 28:360
147. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887
148. Tanaka SC, Schweighofer N, Asahi S, Shishida K, Okamoto Y, Yamawaki S, Doya K (2007) Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS ONE 2:e1333
149. Tolman EC (1938) The determiners of behavior at a choice point. Psychol Rev 45:1
150. Tolman EC (1939) Prediction of vicarious trial and error by means of the schematic sowbug. Psychol Rev 46:318
151. Tolman EC (1948) Cognitive maps in rats and men. Psychol Rev 55:189
152. Tsai HC, Zhang F, Adamantidis A, Stuber GD, Bonci A, de Lecea L, Deisseroth K (2009) Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324:1080
153. Uslaner JM, Acerbo MJ, Jones SA, Robinson TE (2006) The attribution of incentive salience to a stimulus that signals an intravenous injection of cocaine. Behav Brain Res 169:320
154. van der Meer MA, Redish AD (2009) Covert expectation-of-reward in rat ventral striatum at decision points. Frontiers Integr Neurosci 3:1
155. van der Meer MA, Redish AD (2010) Expectancies in decision making, reinforcement learning, and ventral striatum. Front Neurosci 4:29
156. Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43
157. Wagenaar WA (1988) Paradoxes of gambling behavior. Erlbaum, London
158. Weiner I, Lubow RE, Feldon J (1988) Disruption of latent inhibition by acute administration of low doses of amphetamine. Pharmacol Biochem Behav 30:871
159. White AM (2003) What happened? Alcohol, memory blackouts, and the brain. Alcohol Res Health 27(2):186-196
160. Yin HH, Knowlton B, Balleine BW (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur J Neurosci 19:181
161. Yu AJ, Dayan P (2005) Uncertainty, neuromodulation, and attention. Neuron 46:681