Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks

European Journal of Neuroscience

Volumn 28, Issue 8, 2008, Pages 1437-1448

  Yin, Henry H ^a   Ostlund, Sean B ^b   Balleine, Bernard W ^b  

^a DUKE UNIVERSITY   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Basal ganglia; Dopamine; Learning; Nucleus accumbens; Reward; Striatum

Indexed keywords

DOPAMINE;

BASAL GANGLION; CONDITIONING; CORPUS STRIATUM; DOPAMINERGIC NERVE CELL; LEARNING; NONHUMAN; NUCLEUS ACCUMBENS; PRIORITY JOURNAL; REVIEW; REWARD; SENSORIMOTOR CORTEX; VENTRAL TEGMENTUM;

ANIMALS; CEREBRAL CORTEX; CONDITIONING (PSYCHOLOGY); DOPAMINE; HUMANS; LEARNING; NEOSTRIATUM; NEURAL PATHWAYS; NUCLEUS ACCUMBENS; REWARD; VENTRAL TEGMENTAL AREA; VOLITION;

EID: 53949118376     PISSN: 0953816X     EISSN: 14609568     Source Type: Journal    
DOI: 10.1111/j.1460-9568.2008.06422.x     Document Type: Review

References (153)

1. Adams, C.D. (1982) Variations in the sensitivity of instrumental responding to reinforcer devaluation. Q. J. Exp. Psychol., 33b, 109 122.
2. Adams, C.D. Dickinson, A. (1981) Instrumental responding following reinforcer devaluation. Q. J. Exp. Psychol., 33, 109 122.
3. Alexander, G.E., DeLong, M.R. Strick, P.L. (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annu. Rev. Neurosci., 9, 357 381.
4. Ashby, W.R. (1960) Design for a Brain, second Edition. Chapman & Hall, London.
5. Atallah, H.E., Lopez-Paniagua, D., Rudy, J.W. O'Reilly, R.C. (2007) Separate neural substrates for skill learning and performance in the ventral and dorsal striatum. Nat. Neurosci., 10, 126 131.
6. Balleine, B.W. (2001) Incentive processes in instrumental conditioning. In Mowrer, R.R. Klein, S.B. (Eds Handbook of contemporary learning theories. Lawrence Erlbaum Associates, Inc., Publishers, Mahwah, NJ, US, pp. 307 366.
7. Balleine, B.W. (2005) Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol. Behav., 86, 717 730.
8. Balleine, B.W. Dickinson, A. (1998) Goal-directed instrumental action: contingency and incentive learning and their cortical substrates. Neuropharmacology, 37, 407 419.
9. Balleine, B.W. Killcross, S. (2006) Parallel incentive processing: an integrated view of amygdala function. Trends Neurosci., 29, 272 279.
10. Belin, D. Everitt, B.J. (2008) Cocaine Seeking Habits Depend upon Dopamine-Dependent Serial Connectivity Linking the Ventral with the Dorsal Striatum. Neuron, 57, 432 441.
11. Berke, J.D. Hyman, S.E. (2000) Addiction, dopamine, and the molecular mechanisms of memory. Neuron, 25, 515 532.
12. Berridge, K.C. Robinson, T.E. (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Brain Res. Rev., 28, 309 369.
13. Bolles, R. (1972) Reinforcement, expectancy, and learning. Psychol. Rev., 79, 394 409.
14. de Borchgrave, R., Rawlins, J.N., Dickinson, A. Balleine, B.W. (2002) Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats. Exp. Brain Res., 144, 50 68.
15. Brown, P.L. Jenkins, H.M. (1968) Auto-shaping the pigeon's key peck. J. Exp. Anal. Behav., 11, 1 8.
16. Brown, J., Bullock, D. Grossberg, S. (1999) How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci., 19, 10502 10511.
17. Cagniard, B., Beeler, J.A., Britt, J.P., McGehee, D.S., Marinelli, M. Zhuang, X. (2006) Dopamine scales performance in the absence of new learning. Neuron, 51, 541 547.
18. Cardinal, R.N. Cheung, T.H. (2005) Nucleus accumbens core lesions retard instrumental learning and performance with delayed reinforcement in the rat. BMC Neurosci., 6, 9.
19. Cardinal, R.N., Parkinson, J.A., Hall, J. Everitt, B.J. (2002) Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev., 26, 321 352.
20. Cheer, J.F., Aragona, B.J., Heien, M.L., Seipel, A.T., Carelli, R.M. Wightman, R.M. (2007) Coordinated accumbal dopamine release and neural activity drive goal-directed behavior. Neuron, 54, 237 244.
21. Colwill, R.M. Rescorla, R.A. (1986) Associative structures in instrumental learning. In Bower, G. (Ed The Psychology of Learning and Motivation. Academic Press, New York, pp. 55 104.
22. Corbit, L.H. Balleine, B.W. (2003) The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res., 146, 145 157.
23. Corbit, L.H. Janak, P.H. (2007) Inactivation of the lateral but not medial dorsal striatum eliminates the excitatory impact of Pavlovian stimuli on instrumental responding. J. Neurosci., 27, 13977 13981.
24. Corbit, L.H., Muir, J.L. Balleine, B.W. (2001) The role of the nucleus accumbens in instrumental conditioning: Evidence of a functional dissociation between accumbens core and shell. J. Neurosci., 21, 3251 3260.
25. Corbit, L.H., Muir, J.L. Balleine, B.W. (2003) Lesions of mediodorsal thalamus and anterior thalamic nuclei produce dissociable effects on instrumental conditioning in rats. Eur. J. Neurosci., 18, 1286 1294.
26. Corbit, L.H., Janak, P.H. Balleine, B.W. (2007) General and outcome-specific forms of Pavlovian-instrumental transfer: the effect of shifts in motivational state and inactivation of the ventral tegmental area. Eur. J. Neurosci., 26, 3141 3149.
27. Costa, R.M., Cohen, D. Nicolelis, M.A. (2004) Differential corticostriatal plasticity during fast and slow motor skill learning in mice. Curr. Biol., 14, 1124 1134.
28. Cragg, S.J., Hille, C.J. Greenfield, S.A. (2002) Functional domains in dorsal striatum of the nonhuman primate are defined by the dynamic behavior of dopamine. J. Neurosci., 22, 5705 5712.
29. Dalley, J.W., Laane, K., Theobald, D.E., Armstrong, H.C., Corlett, P.R., Chudasama, Y. Robbins, T.W. (2005) Time-limited modulation of appetitive Pavlovian memory by D1 and NMDA receptors in the nucleus accumbens. Proc Natl Acad Sci U S A, 102, 6189 6194.
30. Davis, J. Bitterman, M.E. (1971) Differential reinforcement of other behavior (DRO): A yoked-control comparison. J. Exp. Anal. Behav., 15, 237 241.
31. Day, J.J. Carelli, R.M. (2007) The nucleus accumbens and Pavlovian reward learning. Neuroscientist, 13, 148 159.
32. Day, J.J., Wheeler, R.A., Roitman, M.F. Carelli, R.M. (2006) Nucleus accumbens neurons encode Pavlovian approach behaviors: evidence from an autoshaping paradigm. Eur. J. Neurosci., 23, 1341 1351.
33. Day, J.J., Roitman, M.F., Wightman, R.M. Carelli, R.M. (2007) Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci., 10, 1020 1028.
34. Dayan, P. Balleine, B.W. (2002) Reward, motivation, and reinforcement learning. Neuron, 36, 285 298.
35. Delgado, M.R., Stenger, V.A. Fiez, J.A. (2004) Motivation-dependent responses in the human caudate nucleus. Cereb. Cortex, 14, 1022 1030.
36. Delgado, M.R., Miller, M.M., Inati, S. Phelps, E.A. (2005) An fMRI study of reward-related probability learning. Neuroimage, 24, 862 873.
37. Di Ciano, P., Cardinal, R.N., Cowell, R.A., Little, S.J. Everitt, B.J. (2001) Differential involvement of NMDA, AMPA/kainate, and dopamine receptors in the nucleus accumbens core in the acquisition and performance of pavlovian approach behavior. J. Neurosci., 21, 9471 9477.
38. Dickinson, A. (1985) Actions and habits: the development of behavioural autonomy. Philos. Trans. R. Soc., 308, 67 78.
39. Dickinson, A. (1994) Instrumental Conditioning. In Mackintosh, N.J. (Ed. Animal Learning and Cognition. Academic, Orlando, pp. 45 79.
40. Dickinson, A. Balleine, B. (1993) Actions and responses: The dual psychology of behaviour. In Eilan, N., McCarthy, R.A. Brewer, B. (Eds Spatial Representation: Problems in Philosophy and Psychology. Blackwell Publishers Inc, Malden, MA, US, pp. 277 293.
41. Dickinson, A. Balleine, B. (2002) The role of learning in the operation of motivational systems. In Pashler, H. Gallistel, R. Eds Steven's Handbook of Experimental Psychology (3rd ed.), Vol. 3: Learning, Motivation, and Emotion. John Wiley & Sons, Inc, New York, NY, pp. 497 533.
42. Dickinson, A. Charnock, D.J. (1985) Contingency effects with maintained instrumental reinforcement. Q. J. Exp. Psychol. B, 37, 397 416.
43. Dickinson, A. Dearing, M.F. (1979) Appetitive-aversive interactions and inhibitory processes. In Dickinson, A. Boakes, R.A. (Eds Mechanism of Learning and Motivation. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 203 231.
44. Dickinson, A., Campos, J., Varga, Z.I. Balleine, B. (1996) Bidirectional instrumental conditioning. Q. J. Exp. Psychol B, 49, 289 306.
45. Dickinson, A., Smith, J. Mirenowicz, J. (2000) Dissociation of Pavlovian and instrumental incentive learning under dopamine antagonists. Behav. Neurosci., 114, 468 483.
46. Ellison, G.D. Konorski, J. (1964) Separation of the salivary and motor responses in instrumental conditioning. Science, 146, 1071 1072.
47. Everitt, B.J. Robbins, T.W. (2005) Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat. Neurosci., 8, 1481 1489.
48. Faure, A., Haberland, U., Conde, F. El Massioui, N. (2005) Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. J. Neurosci., 25, 2771 2780.
49. Fiorillo, C.D., Tobler, P.N. Schultz, W. (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science, 299, 1898 1902.
50. Fuster, J.M. (1995). Memory in the Cerebral Cortex. MIT press, Cambridge.
51. Gallistel, C.R., Fairhurst, S. Balsam, P. (2004) The learning curve: implications of a quantitative analysis. Proc. Natl. Acad. Sci. U.S.A., 101, 13124 13131.
52. Goto, Y. Grace, A.A. (2005) Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat. Neurosci., 8, 805 812.
53. Grace, A.A., Floresco, S.B., Goto, Y. Lodge, D.J. (2007) Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends Neurosci., 30, 220 227.
54. Groenewegen, H.J., Berendse, H.W., Wolters, J.G. Lohman, A.H. (1990) The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: evidence for a parallel organization. Prog. Brain Res., 85, 95 116. discussion 116-118.
55. Guthrie, E.R. (1935) The Psychology of Learning. Harpers, New York.
56. Haber, S.N. (2003) The primate basal ganglia: parallel and integrative networks. J. Chem. Neuroanat., 26, 317 330.
57. Haber, S.N., Fudge, J.L. McFarland, N.R. (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. J. Neurosci., 20, 2369 2382.
58. Hammond, L.J. (1980) The effect of contingency upon the appetitive conditioning of free-operant behavior. J. Exp. Anal. Behav., 34, 297 304.
59. Haruno, M. Kawato, M. (2006a) Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Netw, 19, 1242 1254.
60. Haruno, M. Kawato, M. (2006b) Different neural correlates of reward expectation and reward expectation error in the putamen and caudate nucleus during stimulus-action-reward association learning. J. Neurophysiol., 95, 948 959.
61. Haruno, M., Kuroda, T., Doya, K., Toyama, K., Kimura, M., Samejima, K., Imamizu, H. Kawato, M. (2004) A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci., 24, 1660 1665.
62. Hernandez, P.J., Sadeghian, K. Kelley, A.E. (2002) Early consolidation of instrumental learning requires protein synthesis in the nucleus accumbens. Nat. Neurosci., 5, 1327 1331.
63. Hernandez, P.J., Andrzejewski, M.E., Sadeghian, K., Panksepp, J.B. Kelley, A.E. (2005) AMPA/kainate, NMDA, and dopamine D1 receptor function in the nucleus accumbens core: a context-limited role in the encoding and consolidation of instrumental memory. Learn. Mem., 12, 285 295.
64. Hershberger, W.A. (1986) An approach through the looking glass. Anim. Learn. Behav., 14, 443 451.
65. Heyes, C.M. Dawson, G.R. (1990) A demonstration of observational learning in rats using a bidirectional control. Q. J. Exp. Psychol., 42, 59 71.
66. Hikosaka, O., Sakamoto, M. Usui, S. (1989) Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J. Neurophysiol., 61, 814 832.
67. Holland, P.C. (1979) Differential effects of omission contingencies on various components of Pavlovian appetitive conditioned responding in rats. J. Exp. Psychol. Anim. Behav. Process., 5, 178 193.
68. Holland, P.C. (2004) Relations between Pavlovian-instrumental transfer and reinforcer devaluation. J. Exp. Psychol. Anim. Behav. Process., 30, 104 117.
69. Holland, P.C. Rescorla, R.A. (1975) The effect of two ways of devaluing the unconditioned stimulus after first- and second-order appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process., 1, 355 363.
70. Hollerman, J.R., Tremblay, L. Schultz, W. (1998) Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol., 80, 947 963.
71. Houk, J.C., Adams, J.L. Barto, A.G. (1995) A model of how the basal ganglia generates and uses neural signals that predict reinforcement. In Houk, J.C., Davis, J.L. Beiser, D.G. (Eds Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge, MA, pp. 249 270.
72. Hull, C. (1943). Principles of Behavior. Appleton-Century-Crofts, New York.
73. Hyman, S.E., Malenka, R.C. Nestler, E.J. (2006) Neural mechanisms of addiction: the role of reward-related learning and memory. Annu. Rev. Neurosci., 29, 565 598.
74. Jedynak, J.P., Uslaner, J.M., Esteban, J.A. Robinson, T.E. (2007) Methamphetamine-induced structural plasticity in the dorsal striatum. Eur. J. Neurosci., 25, 847 853.
75. Joel, D. Weiner, I. (1994) The organization of the basal ganglia-thalamocortical circuits: open interconnected rather than closed segregated. Neuroscience, 63, 363 379.
76. Joel, D. Weiner, I. (2000) The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum. Neuroscience, 96, 451 474.
77. Jueptner, M., Frith, C.D., Brooks, D.J., Frackowiak, R.S. Passingham, R.E. (1997a) Anatomy of motor learning. II. Subcortical structures and learning by trial and error. J. Neurophysiol., 77, 1325 1337.
78. Jueptner, M., Stephan, K.M., Frith, C.D., Brooks, D.J., Frackowiak, R.S. Passingham, R.E. (1997b) Anatomy of motor learning. I. Frontal cortex and attention to action. J. Neurophysiol., 77, 1313 1324.
79. Kanazawa, I., Murata, M. Kimura, M. (1993) Roles of dopamine and its receptors in generation of choreic movements. Adv. Neurol., 60, 107 112.
80. Kawagoe, R., Takikawa, Y. Hikosaka, O. (1998) Expectation of reward modulates cognitive signals in the basal ganglia. Nat. Neurosci., 1, 411 416.
81. Kimura, M., Aosaki, T. Ishida, A. (1993) Neurophysiological aspects of the differential roles of the putamen and caudate nucleus in voluntary movement. Adv. Neurol., 60, 62 70.
82. Konorski, J. (1967) Integrative Activity of the Brain. University of Chicago Press, Chicago.
83. Lerchner, A., La Camera, G. Richmond, B. (2007) Knowing without doing. Nat. Neurosci., 10, 15 17.
84. Ljungberg, T., Apicella, P. Schultz, W. (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol., 67, 145 163.
85. Lohrenz, T., McCabe, K., Camerer, C.F. Montague, P.R. (2007) Neural signature of fictive learning signals in a sequential investment task. Proc. Natl. Acad. Sci. U.S.A., 104, 9493 9498.
86. Lovibond, P.F. (1983) Facilitation of instrumental behavior by a Pavlovian appetitive conditioned stimulus. J. Exp. Psychol. Anim. Behav. Process., 9, 225 247.
87. Mackintosh, N.J. (1974) The Psychology of Animal Learning. Academic Press, London.
88. Miller, R. (1981) Meaning and Purpose in the Intact Brain. Oxford University Press, New York.
89. Miller, E.K. Cohen, J.D. (2001) An integrative theory of prefrontal cortex function. Annu. Rev. Neurosci., 24, 167 202.
90. Miyachi, S., Hikosaka, O., Miyashita, K., Karadi, Z. Rand, M.K. (1997) Differential roles of monkey striatum in learning of sequential hand movement. Exp. Brain Res., 115, 1 5.
91. Miyachi, S., Hikosaka, O. Lu, X. (2002) Differential activation of monkey striatal neurons in the early and late stages of procedural learning. Exp. Brain Res., 146, 122 126.
92. Montague, P.R., Hyman, S.E. Cohen, J.D. (2004) Computational roles for dopamine in behavioural control. Nature, 431, 760 767.
93. Morris, G., Nevet, A., Arkadir, D., Vaadia, E. Bergman, H. (2006) Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci., 9, 1057 1063.
94. Murschall, A. Hauber, W. (2006) Inactivation of the ventral tegmental area abolished the general excitatory influence of Pavlovian cues on instrumental performance. Learn. Mem., 13, 123 126.
95. Nauta, W.J.H. (1989) Reciprocal links of the corpus striatum with the cerebral cortex and limbic system: A common substrate for movement and thought? In Mueller, J. (Ed Neurology and Psychiatry: A Meeting of Minds. Karger, Basel, pp. 43 63.
96. Nauta, W.J., Smith, G.P., Faull, R.L. Domesick, V.B. (1978) Efferent connections and nigral afferents of the nucleus accumbens septi in the rat. Neuroscience, 3, 385 401.
97. Niv, Y., Daw, N.D., Joel, D. Dayan, P. (2007) Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology (Berl), 191, 507 520.
98. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K. Dolan, R.J. (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304, 452 454.
99. Ostlund, S.B. Balleine, B.W. (2008) Differential involvement of the basolateral amygdala and mediodorsal thalamus in instrumental action selection. J. Neurosci., 28, 4398 4405.
100. Parkinson, J.A., Olmstead, M.C., Burns, L.H., Robbins, T. W. Everitt, B.J. (1999) Dissociation in effects of lesions of the nucleus accumbens core and shell on appetitive pavlovian approach behavior and the potentiation of conditioned reinforcement and locomotor activity by D-amphetamine. J. Neurosci., 19, 2401 2411.
101. Parkinson, J.A., Willoughby, P.J., Robbins, T.W. Everitt, B.J. (2000) Disconnection of the anterior cingulate cortex and nucleus accumbens core impairs Pavlovian approach behavior: further evidence for limbic cortical-ventral striatopallidal systems. Behav. Neurosci., 114, 42 63.
102. Parkinson, J.A., Dalley, J.W., Cardinal, R.N., Bamford, A., Fehnert, B., Lachenal, G., Rudarakanchana, N., Halkerston, K.M., Robbins, T.W. Everitt, B.J. (2002) Nucleus accumbens dopamine depletion impairs both acquisition and performance of appetitive Pavlovian approach behaviour: implications for mesoaccumbens dopamine function. Behav. Brain Res., 137, 149 163.
103. Paxinos, G. Franklin, K. (2003) The Mouse Brain in Stereotaxic Coordinates. Academic Press, New York.
104. Pecina, S., Smith, K.S. Berridge, K.C. (2006) Hedonic hot spots in the brain. Neuroscientist, 12, 500 511.
105. Pithers, R.T. (1985) The roles of event contingencies and reinforcement in human autoshaping and omission responding. Learning and Motivation., 16, 210 237.
106. Pothuizen, H.H., Jongen-Relo, A.L., Feldon, J. Yee, B.K. (2005) Double dissociation of the effects of selective nucleus accumbens core and shell lesions on impulsive-choice behaviour and salience learning in rats. Eur. J. Neurosci., 22, 2605 2616.
107. Rescorla, R.A. (1968) Probability of shock in the presence and absence of CS in fear conditioning. J. Comp. Physiol. Psychol., 66, 1 5.
108. Rescorla, R.A. (1988) Behavioral studies of Pavlovian conditioning. Annu. Rev. Neurosci., 11, 329 352.
109. Rescorla, R.A. Solomon, R.L. (1967) Two-process learning theory: relationships between Pavlovian conditioning and instrumental learning. Psychol. Rev., 74, 151 182.
110. Restle, F. (1957) Discrimination of cues in mazes: a resolution of the "place-vs.-response" question. Psychol. Rev., 64, 217.
111. Reynolds, J.N. Wickens, J.R. (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw., 15, 507 521.
112. Rice, M.E. Cragg, S.J. (2004) Nicotine amplifies reward-related dopamine signals in striatum. Nat. Neurosci., 7, 583 584.
113. Rice, M.E. Cragg, S.J. (2008) Dopamine spillover after quantal release: Rethinking dopamine transmission in the nigrostriatal pathway. Brain Res. Rev., in press doi : Palmiter, R.D. (2007) Viral restoration of dopamine signaling to the dorsal striatum restores instrumental conditioning to dopamine-deficient mice. Psychopharmacology (Berl), 191, 567 578.
114. Roitman, M.F., Wheeler, R.A. Carelli, R.M. (2005) Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output. Neuron, 45, 587 597.
115. Samejima, K., Ueda, Y., Doya, K. Kimura, M. (2005) Representation of action-specific reward values in the striatum. Science, 310, 1337 1340.
116. Schultz, W. (1998a) The phasic reward signal of primate dopamine neurons. Adv. Pharmacol., 42, 686 690.
117. Schultz, W. (1998b) Predictive reward signal of dopamine neurons. J. Neurophysiol., 80, 1 27.
118. Schultz, W., Dayan, P. Montague, P.R. (1997) A neural substrate of prediction and reward. Science, 275, 1593 1599.
119. Schwartz, B. Gamzu, E. (1977) Pavlovian control of operant behavior. In Honig, W. Staddon, J.E.R. (Eds Handbook of Operant Behavior. Prentice Hall, New Jersey, pp. 53 97.
120. Sheffield, F.D. (1965) Relation between classical and instrumental conditioning. In Prokasy, W.F. (Ed Classical Conditioning. Appleton-Century- Crofts, New York, pp. 302 322.
121. Skinner, B. (1938) The Behavior of Organisms. Appleton-Century-Crofts, New York.
122. Smith-Roe, S.L. Kelley, A.E. (2000) Coincident activation of NMDA and dopamine D1 receptors within the nucleus accumbens core is required for appetitive instrumental learning. J. Neurosci., 20, 7737 7742.
123. Sotak, B.N., Hnasko, T.S., Robinson, S., Kremer, E.J. Palmiter, R.D. (2005) Dysregulation of dopamine signaling in the dorsal striatum inhibits feeding. Brain Res., 1061, 88 96.
124. Spence, K. (1947) The role of secondary reinforcement in delayed reward learning. Psychol. Rev., 54, 1 8.
125. Spence, K. (1960) Behavior Theory and Learning. Prentice-Hall, Englewood Cliffs, NJ.
126. Sutton, R.S. Barto, A.G. (1998) Reinforcement Learning. MIT Press, Cambridge.
127. Swanson, L.W. (2000) Cerebral hemisphere regulation of motivated behavior. Brain Res., 886, 113 164.
128. Taha, S.A. Fields, H.L. (2005) Encoding of palatability and appetitive behaviors by distinct neuronal populations in the nucleus accumbens. J. Neurosci., 25, 1193 1202.
129. Taha, S.A. Fields, H.L. (2006) Inhibitions of nucleus accumbens neurons encode a gating signal for reward-directed behavior. J. Neurosci., 26, 217 222.
130. Thorndike, E.L. (1911) Animal Intelligence: Experimental Studies. Macmillan, New York.
131. Tobler, P.N., Dickinson, A. Schultz, W. (2003) Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci., 23, 10402 10410.
132. Tobler, P.N., O'Doherty, J.P., Dolan, R.J. Schultz, W. (2006) Human neural learning depends on reward prediction errors in the blocking paradigm. J. Neurophysiol., 95, 301 310.
133. Tobler, P.N., O'Doherty, J.P., Dolan, R.J. Schultz, W. (2007) Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. J. Neurophysiol., 97, 1621 1632.
134. Trapold, M.A. Overmier, J.B. (1972) The second learning process in instrumental learning. In Black, A.A. Prokasy, W.F. (Eds Classical Conditioning II: Current Research and Theory. Appleton-Century-Crofts, New York, pp. 427 452.
135. Tricomi, E.M., Delgado, M.R. Fiez, J.A. (2004) Modulation of caudate activity by action contingency. Neuron, 41, 281 292.
136. Waelti, P., Dickinson, A. Schultz, W. (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature, 412, 43 48.
137. White, N.M. (1989) A functional hypothesis concerning the striatal matrix and patches: mediation of S-R memory and reward. Life Sci., 45, 1943 1957.
138. Wickens, J.R., Budd, C.S., Hyland, B.I. Arbuthnott, G.W. (2007) Striatal contributions to reward and decision making: making sense of regional variations in a reiterated processing matrix. Ann. N. Y. Acad. Sci., 1104, 192 212.
139. Williams, D.R. (1965) Classical conditioning and incentive motivation. In Prokasy, W.F. (Ed Classical Conditioning. Appleton-Century-Crofts, New York, pp. 340 357.
140. Williams, D.R. Williams, H. (1969) Automaintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J. Exp. Anal. Behav., 12, 511 520.
141. Wiltgen, B.J., Law, M., Ostlund, S., Mayford, M. Balleine, B.W. (2007) The influence of Pavlovian cues on instrumental performance is mediated by CaMKII activity in the striatum. Eur. J. Neurosci., 25, 2491 2497.
142. Wyvell, C.L. Berridge, K.C. (2000) Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: enhancement of reward "wanting" without enhanced "liking" or response reinforcement. J. Neurosci., 20, 8122 8130.
143. Yin, H.H. (2004) The role of the dorsal striatum in goal-directed actions. Doctoral dissertation, Department of Psychology. UCLA, Los Angeles.
144. Yin, H.H. Knowlton, B.J. (2002) Reinforcer devaluation abolishes conditioned cue preference: evidence for stimulus-stimulus associations. Behav. Neurosci., 116, 174 177.
145. Yin, H.H. Knowlton, B.J. (2004) Contributions of striatal subregions to place and response learning. Learn. Mem., 11, 459 463.
146. Yin, H.H. Knowlton, B.J. (2005) Addiction and learning. In Wiers, R.W. Stacy, A. (Eds Handbook of Implicit Cognition and Addiction. Sage, Thousand Oaks, pp. 167 183.
147. Yin, H.H., Knowlton, B.J. Balleine, B.W. (2004) Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. Eur. J. Neurosci., 19, 181 189.
148. Yin, H.H., Knowlton, B.J. Balleine, B.W. (2005a) Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci., 22, 505 512.
149. Yin, H.H., Ostlund, S.B., Knowlton, B.J. Balleine, B.W. (2005b) The role of the dorsomedial striatum in instrumental conditioning. Eur. J. Neurosci., 22, 513 523.
150. Yin, H.H., Knowlton, B.J. Balleine, B.W. (2006a) Inactivation of dorsolateral striatum enhances sensitivity to changes in the action-outcome contingency in instrumental conditioning. Behav. Brain Res., 166, 189 196.
151. Yin, H.H., Zhuang, X. Balleine, B.W. (2006b) Instrumental learning in hyperdopaminergic mice. Neurobiol. Learn. Mem., 85, 283 288.
152. Zahm, D.S. (2000) An integrative neuroanatomical perspective on some subcortical substrates of adaptive responding with emphasis on the nucleus accumbens. Neurosci. Biobehav. Rev., 24, 85 105.
153. Zahm, D.S. (2006) The evolving theory of basal forebrain functional-anatomical 'macrosystems'. Neurosci. Biobehav. Rev., 30, 148 172.