Transition from 'model-based' to 'model-free' behavioral control in addiction: Involvement of the orbitofrontal cortex and dorsolateral striatum

Neuropharmacology

Volumn 76, Issue PART B, 2014, Pages 407-415

  Lucantonio, Federica ^a,b   Caprioli, Daniele ^c,d   Schoenbaum, Geoffrey ^b  

^a UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE   (United States)

^b NATIONAL INSTITUTE ON DRUG ABUSE   (United States)

^c UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^d UK   (United Kingdom)

Author keywords

Addiction; Orbitofrontal cortex; Striatum

Indexed keywords

AMPHETAMINE; COCAINE; PSYCHOSTIMULANT AGENT;

ADDICTION; ARTICLE; BEHAVIOR; BEHAVIOR CONTROL; COCAINE DEPENDENCE; CORPUS STRIATUM; DORSOLATERAL STRIATUM; DRUG SEEKING BEHAVIOR; HABITUAL ADAPTATION; HUMAN; LEARNING; MALADJUSTMENT; NEUROBIOLOGY; NONHUMAN; ORBITAL CORTEX; PRIORITY JOURNAL; REWARD;

ADDICTION; ORBITOFRONTAL CORTEX; STRIATUM;

ANIMALS; COCAINE-RELATED DISORDERS; CORPUS STRIATUM; HUMANS; LEARNING; MODELS, NEUROLOGICAL; PREFRONTAL CORTEX;

EID: 84887031931     PISSN: 00283908     EISSN: 18737064     Source Type: Journal    
DOI: 10.1016/j.neuropharm.2013.05.033     Document Type: Review

References (94)

1. J.K. Alexander, A. Hillier, R.M. Smith, M.E. Tivarus, and D.Q. Beversdorf Beta-adrenergic modulation of cognitive flexibility during stress J. Cogn. Neurosci. 19 2007 468 478 (Pubitemid 46393574)
2. B.W. Balleine, M.R. Delgado, and O. Hikosaka The role of the dorsal striatum in reward and decision-making J. Neurosci. 27 2007 8161 8165 (Pubitemid 47219597)
3. B.W. Balleine, and A. Dickinson Goal-directed instrumental action: contingency and incentive learning and their cortical substrates Neuropharmacology 37 1998 407 419 (Pubitemid 28306930)
4. B.W. Balleine, and J.P. O'Doherty Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action Neuropsychopharmacology 35 2010 48 69
5. K.C. Berridge, T.E. Robinson, and J.W. Aldridge Dissecting components of reward: 'liking', 'wanting', and learning Curr. Opin. Pharmacol. 9 2009 65 73
6. W.J. Brogden Sensory pre-conditioning J. Exp. Psychol. 25 1939 323
7. K.A. Burke, T.M. Franz, D.N. Miller, and G. Schoenbaum The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards Nature 454 2008 340 344
8. K.A. Burke, Y.K. Takahashi, J. Correll, P.L. Brown, and G. Schoenbaum Orbitofrontal inactivation impairs reversal of Pavlovian learning by interfering with 'disinhibition' of responding for previously unrewarded cues Eur. J. Neurosci. 30 2009 1941 1946
9. R.N. Cardinal, J.A. Parkinson, J. Hall, and B.J. Everitt Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex Neurosci. Biobehav. Rev. 26 2002 321 352 (Pubitemid 34555410)
10. S.F. Coffey, G.D. Gudleski, M.E. Saladin, and K.T. Brady Impulsivity and rapid discounting of delayed hypothetical rewards in cocaine-dependent individuals Exp. Clin. Psychopharmacol. 11 2003 18 25 (Pubitemid 36237339)
11. R.M. Colwill, and R.A. Rescorla The role of response-reinforcer associations increases throughout extended instrumental training Anim. Learn. Behav. 16 1988 105 111 (Pubitemid 18046552)
12. L.H. Corbit, and P.H. Janak Inactivation of the lateral but not medial dorsal striatum eliminates the excitatory impact of Pavlovian stimuli on instrumental responding J. Neurosci. 27 2007 13977 13981 (Pubitemid 351377847)
13. H.D. Critchley, and E.T. Rolls Hunger and satiety modify the responses of olfactory and visual neurons in the primate orbitofrontal cortex J. Neurophysiol. 75 1996 1673 1686 (Pubitemid 26114072)
14. N.D. Daw, Y. Niv, and P. Dayan Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control Nat. Neurosci. 8 2005 1704 1711 (Pubitemid 41683198)
15. P. Dayan, and N.D. Daw Decision theory, reinforcement learning, and the brain Cogn. Affect Behav. Neurosci. 8 2008 429 453
16. H. de Wit, and J. Stewart Reinstatement of cocaine-reinforced responding in the rat Psychopharmacology (Berl) 75 1981 134 143 (Pubitemid 12228819)
17. S. de Wit, P.R. Corlett, M.R. Aitken, A. Dickinson, and P.C. Fletcher Differential engagement of the ventromedial prefrontal cortex by goal-directed and habitual behavior toward food pictures in humans J. Neurosci. 29 2009 11330 11338
18. P. Di Ciano Facilitated acquisition but not persistence of responding for a cocaine-paired conditioned reinforcer following sensitization with cocaine Neuropsychopharmacology 33 2008 1426 1431 (Pubitemid 351521729)
19. D.M. Diamond, A.M. Campbell, C.R. Park, J. Halonen, and P.R. Zoladz The temporal dynamics model of emotional memory processing: a synthesis on the neurobiological basis of stress-induced amnesia, flashbulb and traumatic memories, and the Yerkes-Dodson law Neural Plast. 2007 2007 60803
20. A.B. Dickinson, and B.W. Balleine The role of learning in motivation C. Gallistei, Stevens' Handbook of Experimental Psychology 2002 Wiley New York, NY
21. B.B. Doll, D.A. Simon, and N.D. Daw The ubiquity of model-based reinforcement learning Curr. Opin. Neurobiol. 2012
22. K.D. Ersche, J.P. Roiser, T.W. Robbins, and B.J. Sahakian Chronic cocaine but not chronic amphetamine use is associated with perseverative responding in humans Psychopharmacology (Berl) 197 2008 421 431
23. B.J. Everitt, and T.W. Robbins Neural systems of reinforcement for drug addiction: from actions to habits to compulsion Nat. Neurosci. 8 2005 1481 1489 (Pubitemid 41573659)
24. M. Gallagher, R.W. McMahan, and G. Schoenbaum Orbitofrontal cortex and representation of incentive value in associative learning J. Neurosci. 19 1999 6610 6614 (Pubitemid 29347459)
25. S. Ghods-Sharifi, D.M. Haluk, and S.B. Floresco Differential effects of inactivation of the orbitofrontal cortex on strategy set-shifting and reversal learning Neurobiol. Learn. Mem. 89 2008 567 573 (Pubitemid 351484252)
26. J.A. Gottfried, J. O'Doherty, and R.J. Dolan Encoding predictive reward value in human amygdala and orbitofrontal cortex Science 301 2003 1104 1107 (Pubitemid 37022314)
27. A.N. Hampton, P. Bossaerts, and J.P. O'Doherty The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans J. Neurosci. 26 2006 8360 8367 (Pubitemid 44309850)
28. A.N. Hampton, P. Bossaerts, and J.P. O'Doherty Neural correlates of mentalizing-related computations during strategic interactions in humans Proc. Natl. Acad. Sci. U. S. A. 105 2008 6741 6746 (Pubitemid 351754577)
29. C.J. Harmer, and G.D. Phillips Enhanced appetitive conditioning following repeated pretreatment with d-amphetamine Behav. Pharmacol. 9 1998 299 308 (Pubitemid 28380380)
30. C.J. Harmer, and G.D. Phillips Enhanced conditioned inhibition following repeated pretreatment with d-amphetamine Psychopharmacology (Berl) 142 1999 120 131 (Pubitemid 29190846)
31. S.H. Heil, M.W. Johnson, S.T. Higgins, and W.K. Bickel Delay discounting in currently using and currently abstinent cocaine-dependent outpatients and non-drug-using matched controls Addict. Behav. 31 2006 1290 1294
32. P.C. Holland, and J.J. Straub Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning J. Exp. Psychol. Anim. Behav. Process 5 1979 65 78
33. E.W. Holman Some conditions for the dissociation of consummatory and instrumental behavior in rats Learn. Motiv. 6 1975 358 366
34. A. Izquierdo, R.K. Suda, and E.A. Murray Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency J. Neurosci. 24 2004 7540 7548 (Pubitemid 39166745)
35. J.D. Jentsch, P. Olausson, R. De La Garza 2nd, and J.R. Taylor Impairments of reversal learning and response perseveration after repeated, intermittent cocaine administrations to monkeys Neuropsychopharmacology 26 2002 183 190 (Pubitemid 34093081)
36. J.D. Jentsch, and J.R. Taylor Impulsivity resulting from frontostriatal dysfunction in drug abuse: implications for the control of behavior by reward-related stimuli Psychopharmacology (Berl) 146 1999 373 390 (Pubitemid 29513278)
37. M.S. Jog, Y. Kubota, C.I. Connolly, V. Hillegaart, and A.M. Graybiel Building neural representations of habits Science 286 1999 1745 1749 (Pubitemid 129515883)
38. J.L. Jones, G.R. Esber, M.A. McDannald, A.J. Gruber, A. Hernandez, A. Mirenzi, and G. Schoenbaum Orbitofrontal cortex supports behavior and learning using inferred but not cached values Science 338 2012 953 956
39. S. Killcross, and E. Coutureau Coordination of actions and habits in the medial prefrontal cortex of rats Cereb. Cortex 13 2003 400 408 (Pubitemid 36373685)
40. K.N. Kirby, and N.M. Petry Heroin and cocaine abusers have higher discount rates for delayed rewards than alcoholics or non-drug-using controls Addiction 99 2004 461 471 (Pubitemid 38496662)
41. E.F. Kremer The Rescorla-Wagner model: losses in associative strength in compound conditioned stimuli J. Exp. Psychol. Anim. Behav. Process 4 1978 22 36
42. F. Lucantonio, T.A. Stalnaker, Y. Shaham, Y. Niv, and G. Schoenbaum The impact of orbitofrontal dysfunction on cocaine addiction Nat. Neurosci. 15 2012 358 366
43. M.A. McDannald, F. Lucantonio, K.A. Burke, Y. Niv, and G. Schoenbaum Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning J. Neurosci. 31 2011 2700 2705
44. M.A. McDannald, Y.K. Takahashi, N. Lopatina, B.W. Pietras, J.L. Jones, and G. Schoenbaum Model-based learning and the contribution of the orbitofrontal cortex to the model-free world Eur. J. Neurosci. 35 2012 991 996
45. P.R. Montague, P. Dayan, and T.J. Sejnowski A framework for mesencephalic dopamine systems based on predictive Hebbian learning J. Neurosci. 16 1996 1936 1947 (Pubitemid 26145969)
46. A. Nelson, and S. Killcross Amphetamine exposure enhances habit formation J. Neurosci. 26 2006 3805 3812
47. Y. Niv, N.D. Daw, and P. Dayan Choice values Nat. Neurosci. 9 2006 987 988 (Pubitemid 44264904)
48. J. O'Doherty, E.T. Rolls, S. Francis, R. Bowtell, F. McGlone, G. Kobal, B. Renner, and G. Ahne Sensory-specific satiety-related olfactory activation of the human orbitofrontal cortex Neuroreport 11 2000 893 897 (Pubitemid 30173944)
49. J.P. O'Doherty Contributions of the ventromedial prefrontal cortex to goal-directed action selection Ann. N. Y Acad. Sci. 1239 2011 118 129
50. J.P. O'Doherty, P. Dayan, K. Friston, H. Critchley, and R.J. Dolan Temporal difference models and reward-related learning in the human brain Neuron 38 2003 329 337 (Pubitemid 36555657)
51. S.B. Ostlund, and B.W. Balleine Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental conditioning J. Neurosci. 27 2007 4819 4825 (Pubitemid 46699058)
52. J.B. Overmier, J.A. Bull 3rd, and M.A. Trapold Discriminative cue properties of different fears and their role in response selection in dogs J. Comp. Physiol. Psychol. 76 1971 478 482
53. A.M. Owen Cognitive planning in humans: neuropsychological, neuroanatomical and neuropharmacological perspectives Prog. Neurobiol. 53 1997 431 450 (Pubitemid 27509153)
54. P.V. Piazza, and M.L. Le Moal Pathophysiological basis of vulnerability to drug abuse: role of an interaction between stress, glucocorticoids, and dopaminergic neurons Annu. Rev. Pharmacol. Toxicol. 36 1996 359 378 (Pubitemid 26168703)
55. C.L. Pickens, M.P. Saddoris, M. Gallagher, and P.C. Holland Orbitofrontal lesions impair use of cue-outcome associations in a devaluation task Behav. Neurosci. 119 2005 317 322 (Pubitemid 40279521)
56. C.L. Pickens, M.P. Saddoris, B. Setlow, M. Gallagher, P.C. Holland, and G. Schoenbaum Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task J. Neurosci. 23 2003 11078 11084 (Pubitemid 37548942)
57. G.L. Quirarte, I.S. de la Teja, M. Casillas, N. Serafin, R.A. Prado-Alcala, and B. Roozendaal Corticosterone infused into the dorsal striatum selectively enhances memory consolidation of cued water-maze training Learn. Mem. 16 2009 586 589
58. A.D. Redish Addiction as a computational process gone awry Science 306 2004 1944 1947 (Pubitemid 39627860)
59. A.D. Redish, S. Jensen, and A. Johnson A unified framework for addiction: vulnerabilities in the decision process Behav. Brain Sci. 31 2008 415 437
60. T.W. Robbins, and B.J. Everitt Drug addiction: bad habits add up Nature 398 1999 567 570 (Pubitemid 29183674)
61. M.R. Roesch, Y. Takahashi, N. Gugsa, G.B. Bissonette, and G. Schoenbaum Previous cocaine exposure makes rats hypersensitive to both delay and reward magnitude J. Neurosci. 27 2007 245 250 (Pubitemid 46089978)
62. P.H. Rudebeck, and E.A. Murray Amygdala and orbitofrontal cortex lesions differentially influence choices during object reversal learning J. Neurosci. 28 2008 8338 8343
63. M.F. Rushworth, M.P. Noonan, E.D. Boorman, M.E. Walton, and T.E. Behrens Frontal cortex and reward-guided learning and decision-making Neuron 70 2011 1054 1069
64. R. Rygula, S.C. Walker, H.F. Clarke, T.W. Robbins, and A.C. Roberts Differential contributions of the primate ventrolateral prefrontal and orbitofrontal cortex to serial reversal learning J. Neurosci. 30 2010 14552 14559
65. G. Schoenbaum, A.A. Chiba, and M. Gallagher Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning Nat. Neurosci. 1 1998 155 159
66. G. Schoenbaum, A.A. Chiba, and M. Gallagher Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning J. Neurosci. 19 1999 1876 1884 (Pubitemid 29094203)
67. G. Schoenbaum, and B. Setlow Cocaine makes actions insensitive to outcomes but not extinction: implications for altered orbitofrontal-amygdalar function Cereb. Cortex 15 2005 1162 1169 (Pubitemid 41373609)
68. G. Schoenbaum, B. Setlow, S.L. Nugent, M.P. Saddoris, and M. Gallagher Lesions of orbitofrontal cortex and basolateral amygdala complex disrupt acquisition of odor-guided discriminations and reversals Learn. Mem. 10 2003 129 140 (Pubitemid 36418723)
69. G. Schoenbaum, and Y. Shaham The role of orbitofrontal cortex in drug addiction: a review of preclinical studies Biol. Psychiatry 63 2008 256 262
70. L. Schwabe, M.S. Oitzl, C. Philippsen, S. Richter, A. Bohringer, W. Wippich, and H. Schachinger Stress modulates the use of spatial versus stimulus-response learning strategies in humans Learn. Mem. 14 2007 109 116 (Pubitemid 46203573)
71. B. Seymour, J.P. O'Doherty, P. Dayan, M. Koltzenburg, A.K. Jones, R.J. Dolan, K.J. Friston, and R.S. Frackowiak Temporal difference models describe higher-order learning in humans Nature 429 2004 664 667 (Pubitemid 38812318)
72. R.M. Shansky, and J. Lipps Stress-induced cognitive dysfunction: hormone-neurotransmitter interactions in the prefrontal cortex Front. Hum. Neurosci. 7 2013 123
73. M.W. Shiflett The effects of amphetamine exposure on outcome-selective Pavlovian-instrumental transfer in rats Psychopharmacology (Berl) 223 2012 361 370
74. M.W. Shiflett, R.A. Brown, and B.W. Balleine Acquisition and performance of goal-directed instrumental actions depends on ERK signaling in distinct regions of dorsal striatum in rats J. Neurosci. 30 2010 2951 2959
75. N.W. Simon, I.A. Mendez, and B. Setlow Cocaine exposure causes long-term increases in impulsive choice Behav. Neurosci. 121 2007 543 549 (Pubitemid 47445107)
76. R. Sinha How does stress increase risk of drug abuse and relapse? Psychopharmacology (Berl) 158 2001 343 359 (Pubitemid 33141310)
77. T.A. Stalnaker, M.R. Roesch, T.M. Franz, K.A. Burke, and G. Schoenbaum Abnormal associative encoding in orbitofrontal neurons in cocaine-experienced rats during decision-making Eur. J. Neurosci. 24 2006 2643 2653 (Pubitemid 44726776)
78. T.A. Stalnaker, M.R. Roesch, T.M. Franz, D.J. Calu, T. Singh, and G. Schoenbaum Cocaine-induced decision-making deficits are mediated by miscoding in basolateral amygdala Nat. Neurosci. 10 2007 949 951 (Pubitemid 47185199)
79. M. Steinhauser, M. Maier, and R. Hubner Cognitive control under stress: how stress affects strategies of task-set reconfiguration Psychol. Sci. 18 2007 540 545
80. Y. Takahashi, M.R. Roesch, T.A. Stalnaker, and G. Schoenbaum Cocaine exposure shifts the balance of associative encoding from ventral to dorsolateral striatum Front. Integr. Neurosci. 1 2007
81. Y.K. Takahashi, M.R. Roesch, T.A. Stalnaker, R.Z. Haney, D.J. Calu, A.R. Taylor, K.A. Burke, and G. Schoenbaum The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes Neuron 62 2009 269 280
82. J.R. Taylor, and B.A. Horger Enhanced responding for conditioned reward produced by intra-accumbens amphetamine is potentiated after cocaine sensitization Psychopharmacology (Berl) 142 1999 31 40 (Pubitemid 29196061)
83. J.R. Taylor, and J.D. Jentsch Repeated intermittent administration of psychomotor stimulant drugs alters the acquisition of Pavlovian approach behavior in rats: differential effects of cocaine, d-amphetamine and 3,4- methylenedioxymethamphetamine ("Ecstasy") Biol. Psychiatry 50 2001 137 143 (Pubitemid 32728213)
84. S.J. Thorpe, E.T. Rolls, and S. Maddison The orbitofrontal cortex: neuronal activity in the behaving monkey Exp. Brain Res. 49 1983 93 115 (Pubitemid 13180756)
85. L. Tremblay, and W. Schultz Relative reward preference in primate orbitofrontal cortex Nature 398 1999 704 708 (Pubitemid 29197648)
86. E. Tricomi, B.W. Balleine, and J.P. O'Doherty A specific role for posterior dorsolateral striatum in human habit learning Eur. J. Neurosci. 29 2009 2225 2232
87. V.V. Valentin, A. Dickinson, and J.P. O'Doherty Determining the neural substrates of goal-directed learning in the human brain J. Neurosci. 27 2007 4019 4026 (Pubitemid 46597159)
88. M. van der Meer, Z. Kurth-Nelson, and A.D. Redish Information processing in decision-making systems Neuroscientist 18 2012 342 359
89. N.D. Volkow, and J.S. Fowler Addiction, a disease of compulsion and drive: involvement of the orbitofrontal cortex Cereb. Cortex 10 2000 318 325 (Pubitemid 30159298)
90. K. Wunderlich, P. Dayan, and R.J. Dolan Mapping value based planning and extensively trained choice in the human brain Nat. Neurosci. 15 2012 786 791
91. C.L. Wyvell, and K.C. Berridge Incentive sensitization by previous amphetamine exposure: increased cue-triggered "wanting" for sucrose reward J. Neurosci. 21 2001 7831 7840 (Pubitemid 32904983)
92. H.H. Yin, B.J. Knowlton, and B.W. Balleine Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning Eur. J. Neurosci. 19 2004 181 189 (Pubitemid 38122084)
93. H.H. Yin, S.B. Ostlund, B.J. Knowlton, and B.W. Balleine The role of the dorsomedial striatum in instrumental conditioning Eur. J. Neurosci. 22 2005 513 523 (Pubitemid 41097624)
94. A. Zapata, V.L. Minney, and T.S. Shippenberg Shift from goal-directed to habitual cocaine seeking after prolonged experience in rats J. Neurosci. 30 2010 15457 15463