Model-based learning and the contribution of the orbitofrontal cortex to the model-free world

European Journal of Neuroscience

Volumn 35, Issue 7, 2012, Pages 991-996

  Mcdannald, Michael A ^a   Takahashi, Yuji K ^b   Lopatina, Nina ^a   Pietras, Brad W ^a   Jones, Josh L ^a   Schoenbaum, Geoffrey ^a,b  

^a UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE   (United States)

^b NATIONAL INSTITUTE ON DRUG ABUSE   (United States)

Author keywords

Model based; Model free; Orbitofrontal cortex; Pavlovian; Striatum

Indexed keywords

DOPAMINE;

ANIMAL BEHAVIOR; ARTICLE; BLOOD OXYGENATION; BOLD SIGNAL; CONDITIONING; CORPUS STRIATUM; DISCRIMINATION LEARNING; DOPAMINE RELEASE; DOPAMINERGIC NERVE CELL; HUMAN; LEARNING; MESENCEPHALON; MODEL; MODEL BASED PREDICTION ERROR; NONHUMAN; ORBITAL CORTEX; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REWARD; SECOND ORDER CONDITIONING; TEMPORAL DIFFERENCE REINFORCEMENT LEARNING;

ANIMALS; BASAL GANGLIA; FRONTAL LOBE; HUMANS; LEARNING; MODELS, NEUROLOGICAL; NERVE NET;

EID: 84859323549     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2011.07982.x     Document Type: Article

References (57)

1. Balleine, B.W. & Killcross, S. (2006) Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci., 29, 272-279.
2. de Borchgrave, R., Rawlins, J.N., Dickinson, A. & Balleine, B.W. (2002) Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp. Brain Res., 144, 50-68.
3. Brog, J.S., Salyapongse, A., Deutch, A.Y. & Zahm, D.S. (1993) The patterns of afferent innervation of the core and shell in the "accumbens" part of the rat ventral striatum: immunohistochemical detection of retrogradely transported fluoro-gold. J. Comp. Neurol., 338, 255-278.
4. Bromberg-Martin, E.S., Matsumoto, M., Hong, S. & Hikosaka, O. (2010) A pallidus-habenula-dopamine pathway signals inferred stimulus values. J. Neurophysiol., 104, 1068-1076.
5. Burke, K.A., Franz, T.M., Miller, D.N. & Schoenbaum, G. (2008) The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature, 454, 340-344.
6. Corbit, L.H. & Balleine, B.W. (2011) The general and outcome-specific forms of pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. J. Neurosci., 31, 11786-11794.
7. Corbit, L.H., Muir, J.L. & Balleine, B.W. (2001) The role of the nucleus accumbens in instrumental conditioning: evidence of a functional dissociation between accumbens core and shell. J. Neurosci., 21, 3251-3260.
8. D'Ardenne, K., McClure, S.M., Nystrom, L.E. & Cohen, J.D. (2008) BOLD responses reflecting dopaminergic signals in the human ventral tegmental area. Science, 319, 1264-1267.
9. Daw, N.D., Gershman, S.J., Seymour, B., Dayan, P. & Dolan, R.J. (2011) Model-based influences on humans' choices and striatal prediction errors. Neuron, 69, 1204-1215.
10. Day, J.J., Roitman, M.F., Wightman, R.M. & Carelli, R.M. (2007) Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci., 10, 1020-1028.
11. Dayan, P. & Sejnowski, T.J. (1994) Td(Lambda) converges with probability-1. Mach. Learn., 14, 295-301.
12. Delamater, A.R. (2007) The role of the orbitofrontal cortex in sensory-specific encoding of associations in pavlovian and instrumental conditioning. Ann. NY Acad. Sci., 1121, 152-173.
13. Dickinson, A. & Balleine, B.W. (1994) Motivational control of goal-directed action. Anim. Learn. Behav., 22, 1-18.
14. Everitt, B.J., Morris, K.A., O'Brien, A. & Robbins, T.W. (1991) The basolateral amygdala-ventral striatal system and conditioned place preference: further evidence of limbic-striatal interactions underlying reward-related processes. Neuroscience, 42, 1-18.
15. Fiorillo, C.D., Tobler, P.N. & Schultz, W. (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898-1902.
16. Fiorillo, C.D., Newsome, W.T. & Schultz, W. (2008) The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci., 11, 966-973.
17. Gallagher, M., McMahan, R.W. & Schoenbaum, G. (1999) Orbitofrontal cortex and representation of incentive value in associative learning. J. Neurosci., 19, 6610-6614.
18. Gläscher, J., Daw, N., Dayan, P. & O'Doherty, J.P. (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66, 585-595.
19. Haber, S.N., Kunishio, K., Mizobuchi, M. & Lynd-Balta, E. (1995) The orbital and medial prefrontal circuit through the primate basal ganglia. J. Neurosci., 15, 4851-4867.
20. Hall, J., Parkinson, J.A., Connor, T.M., Dickinson, A. & Everitt, B.J. (2001) Involvement of the central nucleus of the amygdala and nucleus accumbens core in mediating Pavlovian influences on instrumental behaviour. Eur. J. Neurosci., 13, 1984-1992.
21. Holland, P.C. (1984) Unblocking in Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process., 10, 476-497.
22. Holland, P.C. & Gallagher, M. (2004) Amygdala-frontal interactions and reward expectancy. Curr. Opin. Neurobiol., 14, 148-155.
23. Holland, P.C. & Rescorla, R.A. (1975) The effect of two ways of devaluing the unconditioned stimulus after first- and second-order appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process., 1, 355-363.
24. Holland, P.C. & Straub, J.J. (1979) Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process., 5, 65-78.
25. Ito, R., Robbins, T.W. & Everitt, B.J. (2004) Differential control over cocaine-seeking behavior by nucleus accumbens core and shell. Nat. Neurosci., 7, 389-397.
26. Izquierdo, A., Suda, R.K. & Murray, E.A. (2004) Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J. Neurosci., 24, 7540-7548.
27. Lapish, C.C., Kroener, S., Durstewitz, D., Lavin, A. & Seamans, J.K. (2007) The ability of the mesocortical dopamine system to operate in distinct temporal modes. Psychopharmacology, 191, 609-625.
28. Lazaro-Munoz, G., LeDoux, J.E. & Cain, C.K. (2010) Sidman instrumental avoidance initially depends on lateral and basal amygdala and is constrained by central amygdala-mediated Pavlovian processes. Biol. Psychiatry, 67, 1120-1127.
29. Lex, B. & Hauber, W. (2010) The role of nucleus accumbens dopamine in outcome encoding in instrumental and Pavlovian conditioning. Neurobiol. Learn. Mem., 93, 283-290.
30. McDannald, M.A., Saddoris, M.P., Gallagher, M. & Holland, P.C. (2005) Lesions of orbitofrontal cortex impair rats' differential outcome expectancy learning but not conditioned stimulus-potentiated feeding. J. Neurosci., 25, 4626-4632.
31. McDannald, M.A., Lucantonio, F., Burke, K.A., Niv, Y. & Schoenbaum, G. (2011) Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci., 31, 2700-2705.
32. Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. (2006) Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci., 9, 1057-1063.
33. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K. & Dolan, R.J. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452-454.
34. Ostlund, S.B. & Balleine, B.W. (2007) Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental conditioning. J. Neurosci., 27, 4819-4825.
35. Pickens, C.L., Saddoris, M.P., Setlow, B., Gallagher, M., Holland, P.C. & Schoenbaum, G. (2003) Different roles for orbitofrontal cortex and basolateral amygdala in a reinforcer devaluation task. J. Neurosci., 23, 11078-11084.
36. Pickens, C.L., Saddoris, M.P., Gallagher, M. & Holland, P.C. (2005) Orbitofrontal lesions impair use of cue-outcome associations in a devaluation task. Behav. Neurosci., 119, 317-322.
37. Ramirez, D.R. & Savage, L.M. (2007) Differential involvement of the basolateral amygdala, orbitofrontal cortex, and nucleus accumbens core in the acquisition and use of reward expectancies. Behav. Neurosci., 121, 896-906.
38. Rescorla, R.A. (1988) Pavlovian conditioning. It's not what you think it is. Am. Psychol., 43, 151-160.
39. Rescorla, R.A. (1999) Learning about qualitatively different outcomes during a blocking procedure. Anim. Learn. Behav., 27, 140-151.
40. Rescorla, R.A. & Wagner, A.R. (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In Black, A.H. & Prokasy, W.F. (Eds), Classical Conditioning II: Current Research and Theory. Appleton Century Crofts, New York, pp. 64-99.
41. Roesch, M.R., Calu, D.J. & Schoenbaum, G. (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci., 10, 1615-1624.
42. Roitman, M.F., Wheeler, R.A., Wightman, R.M. & Carelli, R.M. (2008) Real-time chemical responses in the nucleus accumbens differentiate rewarding and aversive stimuli. Nat. Neurosci., 11, 1376-1377.
43. Saddoris, M.P., Stamatakis, A. & Carelli, R.M. (2011) Neural correlates of Pavlovian-to-instrumental transfer in the nucleus accumbens shell are selectively potentiated following cocaine self-administration. Eur. J. Neurosci., 33, 2274-2287.
44. Schoenbaum, G. & Setlow, B. (2005) Cocaine makes actions insensitive to outcomes but not extinction: implications for altered orbitofrontal-amygdalar function. Cereb. Cortex, 15, 1162-1169.
45. Schoenbaum, G., Saddoris, M.P., Ramus, S.J., Shaham, Y. & Setlow, B. (2004) Cocaine-experienced rats exhibit learning deficits in a task sensitive to orbitofrontal cortex lesions. Eur. J. Neurosci., 19, 1997-2002.
46. Schultz, W., Apicella, P., Scarnati, E. & Ljungberg, T. (1992) Neuronal activity in monkey ventral striatum related to the expectation of reward. J. Neurosci., 12, 4595-4610.
47. Schultz, W., Dayan, P. & Montague, P.R. (1997) A neural substrate of prediction and reward. Science, 275, 1593-1599.
48. Setlow, B., Holland, P.C. & Gallagher, M. (2002) Disconnection of the basolateral amygdala complex and nucleus accumbens impairs appetitive pavlovian second-order conditioned responses. Behav. Neurosci., 116, 267-275.
49. Singh, T., McDannald, M.A., Haney, R.Z., Cerri, D.H. & Schoenbaum, G. (2010) Nucleus accumbens core and shell are necessary for reinforcer devaluation effects on pavlovian conditioned responding. Front. Integr. Neurosci., 4, 126.
50. Stuber, G.D., Hnasko, T.S., Britt, J.P., Edwards, R.H. & Bonci, A. (2010) Dopaminergic terminals in the nucleus accumbens but not the dorsal striatum corelease glutamate. J. Neurosci., 30, 8229-8233.
51. Sutton, R.S. (1988) Learning to predict by the method of temporal difference. Mach. Learn., 3, 9-44.
52. Takahashi, Y.K., Roesch, M.R., Wilson, R.C., Toreson, K., O'Donnell, P., Niv, Y. & Schoenbaum, G. (2011) Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci.. 14, 1590-1597.
53. Tolman, E. (1948) Cognitive maps in rats and men. Psychol. Rev., 55, 189-208.
54. Trapold, M.A. & Overmier, J.B. (1972) The second learning process in isntrumental training. In Black, A. & Prokasy, W.F. (Eds), Classical Conditioning II. Appleton-Century-Crofts, New York, pp. 427-452.
55. Walton, M.E., Behrens, T.E.J., Buckley, M.J., Rudebeck, P.H. & Rushworth, M.F.S. (2010) Separable learning systems in the macaque brain and the role of orbitofrontal cortex in contingent learning. Neuron, 65, 927-939.
56. Wyvell, C.L. & Berridge, K.C. (2001) Incentive sensitization by previous amphetamine exposure: increased cue-triggered "wanting" for sucrose reward. J. Neurosci., 21, 7831-7840.
57. Xin, J. & Costa, R.M. (2010) Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466, 457-462.