Learning and reversal learning in the subcortical limbic system: A computational model

Adaptive Behavior

Volumn 18, Issue 3, 2010, Pages 211-236

  Thompson, Adedoyin Maria ^a   Porr, Bernd ^a   Wörgötter, Florentin ^b  

^a UNIVERSITY OF GLASGOW   (United Kingdom)

^b Bernstein Center for Computational Neuroscience   (Germany)

Author keywords

dopamine; nucleus accumbens; reversal learning; three factor ISO Learning

Indexed keywords

ANIMALIA;

EID: 77955762231     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712309353612     Document Type: Article

References (95)

1. Alcaro, A., Huber, R., & Panksepp, J. (2007). Behavioral functions of the mesolimbic dopaminergic system: an affective neuroethological perspective. Brain Research Reviews, 56(2), 283-321.
2. Alheid, G.F., & Heimer, L. (1988). New perspectives in basal forebrain organization of special relevance for neuropsychiatric disorders: the striatopallidal, amygdaloid, and corticopetal components of substantia innominata. Neuroscience, 27, 1-39.
3. Balleine, B., & Killcross, S. (1994). Effects of ibotenic acid lesions of the nucleus accumbens on instrumental action. Behavioural Brain Research, 65(2), 181-193.
4. Barrot, M., Marinelli, M., Abrous, D.N., Rougé-Pont, F., Le Moal, M., & Piazza, P.V. (2000). The dopaminergic hyper-responsiveness of the shell of the nucleus accumbens is hormone-dependent. European Journal of Neuroscience, 12(3), 973-979.
5. Birrell, J.M., & Brown, V.J. (2000). Medial frontal cortex mediates perceptual attentional set shifting in the rat. Journal of Neuroscience, 20(11), 4320-4324.
6. Blaiss, C.A., & Janak, P.H. (2009). The nucleus accumbens core and shell are critical for the expression, but not the consolidation, of Pavlovian conditioned approach. Behavioural Brain Research, 200(1), 22-32.
7. Bouton, M.E. (2002). Context, ambiguity, and unlearning: sources of relapse after behavioral extinction. Biological Psychiatry, 52(10), 976-986.
8. Braitenberg, V. (1984). Vehicles: Explorations in synthetic psychology. Cambridge, MA: MIT Press.
9. Brog, J.S., Salyapongse, A., Deutch, A.Y., & Zahm, D.S. (1993). The patterns of afferent innervation of the core and shell in the accumbens part of the rat ventral striatum: immunohistochemical detection of retrogradely transported fluoro-gold. Journal of Comparative Neurology, 338(2), 255-278.
10. Bushnell, P.J., & Stanton, M.E. (1991). Serial spatial reversal learning in rats: comparison of instrumental and auto-maintenance procedures. Physiology & Behavior, 50(6), 1145-1151.
11. Calabresi, P., Maj, R., Mercuri, N.B., & Bernardi, G. (1992). Coactivation of d1 and d2 dopamine receptors is required for long-term synaptic depression in the striatum. Neuroscience Letters, 142(1), 95-99.
12. Calabresi, P., Maj, R., Pisani, A., Mercuri, N.B., & Bernardi, G. (1992). Long-term synaptic depression in the striatum: physiological and pharmacological characterization. Journal of Neuroscience, 12(11), 4224-4233.
13. Calabresi, P., Picconi, B., Tozzi, A., & Di Filippo, M. (2007). Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends in Neuroscience, 30(5), 211-219.
14. Calabresi, P., Pisani, A., Mercuri, N.B., & Bernardi, G. (1996). The corticostriatal projection: from synaptic plasticity to dysfunctions of the basal ganglia. Trends in Neuroscience, 19(1), 19-24.
15. Calaminus, C., & Hauber, W. (2007). Intact discrimination reversal learning but slowed responding to reward-predictive cues after dopamine d1 and d2 receptor blockade in the nucleus accumbens of rats. Psychopharmacology (Berlin), 191(3), 551-566.
16. Calaminus, C., & Hauber, W. (2008). Guidance of instrumental behavior under reversal conditions requires dopamine d1 and d2 receptor activation in the orbitofrontal cortex. Neuroscience, 154(4), 1195-1204.
17. Cardinal, R.N., Parkinson, J.A., Hall, J., & Everitt, B.J. (2002). Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience & Biobehavioral Reviews, 26(3), 321-352.
18. Cardinal, R.N., Parkinson, J.A., Lachenal, G., Halkerston, K.M., Rudarakanchana, N., Hall, J.,et al.(2002). Effects of selective excitotoxic lesions of the nucleus accumbens core, anterior cingulate cortex, and central nucleus of the amygdala on autoshaping performance in rats. Behavioral Neuroscience, 116(4), 553-567.
19. Chudasama, Y., Bussey, T.J., & Muir, J.L. (2001). Effects of selective thalamic and prelimbic cortex lesions on two types of visual discrimination and reversal learning. European Journal of Neuroscience, 14(6), 1009-1020.
20. Cools, R., Lewis, S.J., Clark, L., Barker, R.A., & Robbins, T.W. (2007). L-dopa disrupts activity in the nucleus accumbens during reversal learning in Parkinson's disease. Neuropsychopharmacology, 32(1), 180-189.
21. Corbit, L.H., Muir, J.L., & Balleine, B.W. (2001). The role of the nucleus accumbens in instrumental conditioning: Evidence of a functional dissociation between accumbens core and shell. Journal of Neuroscience, 21(9), 3251- 3260.
22. Cragg, S.J. (2006). Meaningful silences: how dopamine listens to the ach pause. Trends in Neuroscience, 29(3), 125-131.
23. Creese, I., Sibley, D.R., & Leff, S. (1983). Classification of dopamine receptors. Advances in Biochemical Psychopharmacology, 37, 255-266.
24. Daw, N.D., Kakade, S., & Dayan, P. (2002). Opponent interactions between serotonin and dopamine. Neural Networks, 15(4-6), 603-616.
25. Dayan, P. (2001). Motivated reinforcement learning. In T. K. Leen, T. G. Dietterich & V. Tresp Advances in Neural Information Processing Systems 14. Cambridge, MA: MIT Press (pp. 11-18).
26. Di Ciano, P., Cardinal, R.N., Cowell, R.A., Little, S.J., & Everitt, B.J. (2001). Differential involvement of nmda, ampa/kainate, and dopamine receptors in the nucleus accumbens core in the acquisition and performance of Pavlovian approach behavior. Journal of Neuroscience, 21(23), 9471-9477.
27. Durstewitz, D., & Seamans, J.K. (2002). The computational role of dopamine d1 receptors in working memory. Neural Networks, 15(4-6), 561-572.
28. Egerton, A., Brett, R.R., & Pratt, J.A. (2005). Acute delta9-tetrahydrocannabinol-induced deficits in reversal learning: neural correlates of affective inflexibility. Neuropsychopharmacology, 30(10), 1895-1905.
29. Fiorillo, C.D., Tobler, P.N., & Schultz, W. (2003). Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299(5614), 1898-1902.
30. Floresco, S.B., McLaughlin, R.J., & Haluk, D.M. (2008). Opposing roles for the nucleus accumbens core and shell in cue-induced reinstatement of food-seeking behavior. Neuroscience, 154(3), 877-884.
31. Floresco, S.B., West, A.R., Ash, B., Moore, H., & Grace, A.A. (2003). Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nature Neuroscience, 6(9), 968-973.
32. Funahashi, S., Bruce, C.J., & Goldman-Rakic, P.S. (1989). Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. Journal of Neurophysiology, 61(2), 331-349.
33. Ghitza, U.E., Fabbricatore, A.T., Prokopenko, V., Pawlak, A.P., & West, M.O. (2003). Persistent cue-evoked activity of accumbens neurons after prolonged abstinence from self-administered cocaine. Journal of Neuroscience, 23(19), 7239-7245.
34. Gonon, F. (1997). Prolonged and extrasynaptic excitatory action of dopamine mediated by d1 receptors in the rat striatum in vivo. Journal of Neuroscience, 17(15), 5972- 5978.
35. Gonon, F., & Sundstrom, L. (1996). Excitatory effects of dopamine released by impulse flow in the rat nucleus accumbens in vivo. Neuroscience, 75(1), 13-18.
36. Goto, Y., & Grace, A.A. (2005). Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nature Neuroscience, 8(6), 805-812.
37. Goto, Y., & Grace, A.A. (2008). Limbic and cortical information processing in the nucleus accumbens. Trends in Neuroscience, 31(11), 552-558.
38. Grace, A.A. (1991). Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience, 41(1), 1-24.
39. Grace, A.A. (2000). The tonic/phasic model of dopamine system regulation and its implications for understanding alcohol and psychostimulant craving. Addiction, 95 (Suppl 2), 119-128.
40. Grace, A.A., Floresco, S.B., Goto, Y., & Lodge, D.J. (2007). Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends in Neuroscience, 30(5), 220-227.
41. Groenewegen, H.J., Galis-de Graaf, Y., & Smeets, W.J. (1999). Integration and segregation of limbic cortico-striatal loops at the thalamic level: an experimental tracing study in rats. Journal of Chemical Neuroanatomy, 16(3), 167-185.
42. Horvitz, J.C. (2000). Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience, 96(4), 651-656.
43. Ishikawa, A., Ambroggi, F., Nicola, S.M., & Fields, H.L. (2008). Contributions of the amygdala and medial prefrontal cortex to incentive cue responding. Neuroscience, 155(3), 573-584.
44. Ito, R., Robbins, T.W., & Everitt, B.J. (2004). Differential control over cocaine-seeking behavior by nucleus accumbens core and shell. Nature Neuroscience, 7(4), 389-397.
45. Kelley, A.E. (1999). Functional specificity of ventral striatal compartments in appetitive behaviors. Annals of the New York Academy of Sciences, 877, 71-90.
46. Kelley, A.E. (2004). Ventral striatal control of appetitive motivation: role in ingestive behavior and reward-related learning. Neuroscience & Biobehavioral Reviews, 27(8), 765-776.
47. Law-Tho, D., Desce, J.M., & Crepel, F. (1995). Dopamine favours the emergence of long-term depression versus long-term potentiation in slices of rat prefrontal cortex. Neuroscience Letters, 188(2), 125-128.
48. Lovinger, D.M., Partridge, J.G., & Tang, K.C. (2003). Plastic control of striatal glutamatergic transmission by ensemble actions of several neurotransmitters and targets for drugs of abuse. Annals of the New York Academy of Sciences, 1003, 226-240.
49. Maeno, H. (1982). Dopamine receptors in canine caudate nucleus. Molecular and Cellular Biochemistry, 43(2), 65- 80.
50. Malenka, R.C., & Bear, M.F. (2004). Ltp and ltd: an embarrassment of riches. Neuron, 44(1), 5-21.
51. Matsuda, Y., Marzo, A., & Otani, S. (2006). The presence of background dopamine signal converts long-term synaptic depression to potentiation in rat prefrontal cortex. Journal of Neuroscience, 26(18), 4803-4810.
52. McKittrick, C.R., & Abercrombie, E.D. (2007). Catecholamine mapping within nucleus accumbens: differences in basal and amphetamine-stimulated efflux of norepinephrine and dopamine in shell and core. Journal of Neuro-chemistry, 100(5), 1247-1256.
53. Means, L.W., Hershey, A.E., Waterhouse, G.J., & Lane, C.J. (1975). Effects of dorsomedial thalamic lesions on spatial discrimination reversal in the rat. Physiology & Behavior, 14(6), 725-729.
54. Mogenson, G.J., Jones, D.L., & Yim, C.Y. (1980). From motivation to action: functional interface between the limbic system and the motor system. Progress in Neurobiology, 14(2-3), 69-97.
55. Napier, R.M., Macrae, M., & Kehoe, E.J. (1992). Rapid reaquisition in conditioning of the rabbit's nictitating membrane response. Journal of Experimental Psychology: Animal Behavior Processes, 18(2), 182-192.
56. Nicola, S.M., Woodward Hopf, F., & Hjelmstad, G.O. (2004). Contrast enhancement: a physiological effect of striatal dopamine? Cell Tissue Research, 318(1), 93-106.
57. Niv, Y., Daw, N.D., Joel, D., & Dayan, P. (2007). Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berlin), 191(3), 507-520.
58. Olds, J., & Milner, P. (1954). Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative Physiological Psychology, 47(6), 419-427.
59. O'Reilly, R.C., Frank, M.J., Hazy, T.E., & Watz, B. (2007). Pvlv: the primary value and learned value Pavlovian learning algorithm. Behavioral Neuroscience, 121(1), 31- 49.
60. Papp, M., & Bal, A. (1987). Separation of the motivational and motor consequences of 6-hydroxydopamine lesions of the mesolimbic or nigrostriatal system in rats. Behavioural Brain Research, 23(3), 221-229.
61. Parkinson, J.A., Cardinal, R.N., & Everitt, B.J. (2000). Limbic cortical-ventral striatal systems underlying appetitive conditioning. Progress in Brain Research, 126, 263-285.
62. Parkinson, J.A., Olmstead, M.C., Burns, L.H., Robbins, T.W., & Everitt, B.J. (1999). Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive Pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by d-amphetamine. Journal of Neuroscience, 19(6), 2401- 2411.
63. Parkinson, J.A., Willoughby, P.J., Robbins, T.W., & Everitt, B.J. (2000). Disconnection of the anterior cingulate cortex and nucleus accumbens core impairs Pavlovian approach behavior: further evidence for limbic cortical-ventral striatopallidal systems. Behavioral Neuroscience, 114(1), 42-63.
64. Passetti, F., Chudasama, Y., & Robbins, T.W. (2002). The frontal cortex of the rat and visual attentional performance: dissociable functions of distinct medial prefrontal subregions. Cerebral Cortex, 12(12), 1254-1268.
65. Pavlov, I.P. (1927). Conditioned reflexes. Oxford: Oxford University Press.
66. Pawlak, V., & Kerr, J.N. (2008). Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. Journal of Neuroscience, 28(10), 2435-2446.
67. Peters, J., LaLumiere, R.T., & Kalivas, P.W. (2008). Infralimbic prefrontal cortex is responsible for inhibiting cocaine seeking in extinguished rats. Journal of Neuroscience, 28(23), 6046-6053.
68. Phillips, A.G., Brooke, S.M., & Fibiger, H.C. (1975). Effects of amphetamine isomers and neuroleptics on self-stimulation from the nucleus accumbens and dorsal noradrenergic bundle. Brain Research, 85(1), 13-22.
69. Phillips, G.D., Robbins, T.W., & Everitt, B.J. (1994). Bilateral intra-accumbens self-administration of d-amphetamine: antagonism with intra-accumbens sch-23390 and sulpiride. Psychopharmacology (Berlin), 114(3), 477-485.
70. Porr, B., & Wörgötter, F. (2003). Isotropic Sequence Order learning. Neural Computation, 15, 831-864.
71. Porr, B., & Wörgötter, F. (2007). Learning with relevance: Using a third factor to stabilise Hebbian learning. Neural Computation. 19(10), 2694-2719.
72. Prescott, T.J., Montes González, F.M., Gurney, K., Humphries, M.D., & Redgrave, P. (2006). A robot model of the basal ganglia: Behavior and intrinsic processing. Neural Networks, 19(1), 31-61.
73. Rescorla, R.A. (2001). Experimental extinction. In S. B. Klein & R. R. Mowrer (Eds), Handbook of contemporary learning theories (pp. 119-154). Mahwah, NJ: Lawrence Erlbaum Associates.
74. Reynolds, J.N., & Wickens, J.R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15(4-6), 507-521.
75. Reynolds, S.M., & Berridge, K.C. (2001). Fear and feeding in the nucleus accumbens shell: rostrocaudal segregation of gaba-elicited defensive behavior versus eating behavior. Journal of Neuroscience, 21(9), 3261-3270.
76. Robbins, T.W., & Everitt, B.J. (1996). Neurobehavioural mechanisms of reward and motivation. Current Opinion in Neurobiology, 6(2), 228-236.
77. Roberts, D.C., Corcoran, M.E., & Fibiger, H.C. (1977). On the role of ascending catecholaminergic systems in intravenous self-administration of cocaine. Pharmacology, Biochemistry and Behavior, 6(6), 615-620.
78. Salamone, J.D., Cousins, M.S., & Snyder, B.J. (1997). Behavioral functions of nucleus accumbens dopamine: empirical and conceptual problems with the anhedonia hypothesis. Neuroscience & Biobehavioral Reviews, 21(3), 341-359.
79. Schotanus, S.M., & Chergui, K. (2008). Dopamine d1 receptors and group i metabotropic glutamate receptors contribute to the induction of long-term potentiation in the nucleus accumbens. Neuropharmacology, 54(5), 837-844.
80. Schultz, W. (1997). Dopamine neurons and their role in reward mechanisms. Current Opinion in Neurobiology, 7(2), 191-197.
81. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1-27.
82. Sutton, R., & Barto, A. (1982). Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element. Behavioural Brain Research, 4(3), 221-235.
83. Sutton, R.S., & Barto, A. (1987). A temporal-difference model of classical conditioning. In Proceedings of the Ninth Annual Conference of the Cognitive Science Society (pp. 355-378), Seattle, Washington.
84. Sutton, R.S., & Barto, A. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. Moore (Eds), Learning and computational neuroscience (pp. 497-537). Cambridge, MA: MIT-press.
85. Sutton, R.S., & Barto, A.G. (1998). Reinforcement learning: an introduction. Cambridge, MA: MIT.
86. Thompson, A., Porr, B., Kolodziejsky, C., & Wörgötter, F. (2008). Second order conditioning in the sub-cortical nuclei of the limbic system. In M. Asada (Ed.) Proceedings of the Tenth International Conference on Simulation of Adaptive Behavior, SAB, LNAI 5040 (pp. 189-198). New York: Springer.
87. Thompson, A., Porr, B., & Wörgötter, F. (2006). Stabilising Hebbian learning with a third factor in a food retrieval task. In S. Nolfi (Ed.), Proceedings of the Ninth International Conference on Simulation of Adaptive Behavior, SAB, LNAI 4095 (pp. 313-322). New York: Springer.
88. Verschure, P.F., Voegtlin, T., & Douglas, R.J. (2003). Environmentally mediated synergy between perception and behaviour in mobile robots. Nature, 425(6958), 620-624.
89. Wise, R.A. (1998). Drug-activation of brain reward pathways. Drug and Alcohol Dependence, 51(1-2), 13-22.
90. Wise, R.A. (2004). Dopamine, learning and motivation. Nature Reviews Neuroscience, 5(6), 483-494.
91. Wise, R.A., & Rompre, P.P. (1989). Brain dopamine and reward. Annual Review of Psychology, 40, 191-225.
92. Wise, R.A., Spindler, J., deWit, H., & Gerberg, G.J. (1978). Neuroleptic-induced anhedonia in rats: pimozide blocks reward quality of food. Science, 201(4352), 262-264.
93. Zahm, D.S. (2000). An integrative neuroanatomical perspective on some subcortical substrates of adaptive responding with emphasis on the nucleus accumbens. Neuroscience & Biobehavioral Reviews, 24(1), 85-105.
94. Zahm, D.S., & Brog, J.S. (1992). On the significance of sub-territories in the accumbens part of the rat ventral striatum. Neuroscience, 50(4), 751-767.
95. Zahm, D.S., & Heimer, L. (1990). Two transpallidal pathways originating in the rat nucleus accumbens. Journal of Comparative Neurology, 302(3), 437-446.