Dopamine neurons and their role in reward mechanisms

Current Opinion in Neurobiology

Volumn 7, Issue 2, 1997, Pages 191-197

  Schultz, Wolfram ^a  

^a UNIVERSITY OF FRIBOURG   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

BEHAVIOR; DOPAMINERGIC NERVE CELL; DOPAMINERGIC TRANSMISSION; LEARNING; NONHUMAN; PRIORITY JOURNAL; REVIEW; REWARD; SYNAPSE;

EID: 0031005828     PISSN: 09594388     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0959-4388(97)80007-4     Document Type: Article

References (83)

1. Wise RA. Neuroleptics and operant behavior: the anhedonia hypothesis. Behav Brain Sci. 5:1982;39-87.
2. Beninger RJ. The role of dopamine in locomotor activity and learning. Brain Res Rev. 6:1983;173-196.
3. Fibiger HC, Phillips AG. Reward, motivation, cognition: psychobiology of mesotelencephalic dopamine systems. Bloom FE. Handbook of Physiology - The Nervous System IV. 1986;647-675 Williams and Wilkins, Baltimore.
4. Amalric M, Koob GF. Depletion of dopamine in the caudate nucleus but not in nucleus accumbens in impairs reaction-time performance. J Neurosci. 7:1987;2129-2134.
5. Robbins TW, Everitt BJ. Functions of dopamine in the dorsal and ventral striatum. Semin Neurosci. 4:1992;119-128.
6. Fibiger HC. Mesolimbic dopamine: an analysis of its role in motivated behavior. Semin Neurosci. 5:1993;321-327.
7. of outstanding interest Robbins TW, Everitt BJ. Neurobehavioral mechanisms of reward and motivation. Curr Opin Neurobiol. 6:1996;228-236 An excellent review combining conceptual elaborations of reward functions with neurobiological correlates in limbic and basal ganglia structures. Together with a previous review [5], it also gives a detailed overview of many years of the authors' experimental and conceptual work dealing with the role of dopamine systems in behavior.
8. Wise RA, Bozarth MA. A psychomotor stimulant theory of addiction. Psychol Rev. 94:1987;469-492.
9. Koob GF. Dopamine, addiction and reward. Semin Neurosci. 4:1992;139-148.
10. Robinson TE, Berridge KC. The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res Rev. 18:1993;247-291.
11. of outstanding interest Di Chiara G. The role of dopamine in drug abuse viewed from the perspective of its role in motivation. Drug Alcohol Depend. 38:1995;95-137 A systematic review on the involvement of dopamine systems in drug addiction. Following an extensive overview of many aspects of dopamine's function in approach behavior, reward and motivation, detailed mechanisms of drug action on dopamine systems are described in chapters dealing with drug self-administration, discrimination, sensitization and dependence.
12. of outstanding interest Wise RA. Neurobiology of addiction. Curr Opin Neurobiol. 6:1996;243-251 A concentrated overview of addiction and the brain structures involved. The extensive reviewing activity of the author (see e.g. [1,8,14]) provides for stimulating reading on experimentation and conceptualization of most aspects of drug abuse, reward functions and dopamine mechanisms.
13. Konorski J. Conditioned Reflexes and Neuron Organization. 1948;Cambridge University Press, Cambridge.
14. Wise RA. The brain and reward. LIeberman JM, Cooper SJ. The Neuropharmacological Basis of Reward. 1989;377-424 Oxford University Press, Oxford.
15. White NW, Milner PM. The psychobiology of reinforcers. Annu Rev Psychol. 43:1992;443-471.
16. Dickinson A, Balleine B. Motivational control of goal-directed action. Anim Learning Behav. 22:1994;1-18.
17. Dickinson A. Contemporary Animal Learning Theory. 1980;Cambridge University Press, Cambridge.
18. Rescorla RA, Wagner AR. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Black AH, Prokasy WF. Classical Conditioning II: Current Research and Theory. 1972;64-99 Appleton Century Crofts, New York.
19. Mackintosh NJ. A theory of attention: variations in the associability of stimulus with reinforcement. Psychol Rev. 82:1975;276-298.
20. Pearce JM, Hall GA. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol Rev. 87:1980;532-552.
21. Romo R, Schultz W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol. 63:1990;592-606.
22. Schultz W, Romo R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J Neurophysiol. 63:1990;607-624.
23. Ljungberg T, Apicella P, Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol. 67:1992;145-163.
24. Mirenowicz J, Schultz W. Importance of unpredictedness for reward responses in primate dopamine neurons. J Neurophysiol. 72:1994;1024-1027.
25. Miller JD, Sanghera MK, German DC. Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sci. 29:1981;1255-1263.
26. Schultz W. Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J Neurophysiol. 56:1986;1439-1462.
27. of special interest Mirenowicz J, Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature. 379:1996;449-451 The results presented in this paper suggest that only a small fraction of probably less selective dopamine neurons respond to non-noxious, but clearly aversive, stimuli, whereas the majority of dopamine neurons are activated by appetitive stimuli. The study also describes conditions of response generalization with non-appetitive stimuli, revealing a probably attentional component of appetitive dopamine neuron responses.
28. Steinfels GF, Heym J, Strecker RE, Jacobs BL. Behavioral correlates of dopaminergic unit activity in freely moving cats. Brain Res. 258:1983;217-228.
29. Strecker RE, Jacobs BL. Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res. 361:1985;339-350.
30. Nishino H, Ono T, Muramoto KI, Fukuda M, Sasaki K. Neuronal activity in the ventral tegmental area (VTA) during motivated bar press feeding behavior in the monkey. Brain Res. 413:1987;302-313.
31. Kosobud AEK, Harris GC, Chapin JK. Behavioral associations of neuronal activity in the ventral tegmental area of the rat. J Neurosci. 14:1994;7117-7129.
32. Schultz W, Ruffieux A, Aebischer P. The activity of pars compacta neurons of the monkey substantia nigra in relation to motor activation. Exp Brain Res. 51:1983;377-387.
33. DeLong MR, Crutcher MD, Georgopoulos AP. Relations between movement and single cell discharge in the substantia nigra of the behaving monkey. J Neurosci. 3:1983;1599-1606.
34. Schultz W, Apicella P, Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci. 13:1993;900-913.
35. Schultz W, Romo R, Ljungberg T, Mirenowicz J, Hollerman JR, Dickinson A. Reward-related signal carried by dopamine neurons. Houk JC, Davis JL, Beiser DG. Models of Informaiton Processing in the Basal Ganglia. 1995;233-248 MIT Press, Cambridge, MA.
36. Scarnati E, Proia A, Campana E, Pacitti C. A microiontophoretic study on the nature of the putative synaptic neurotransmitter involved in the pedunculopontine-substantia nigra pars compacta excitatory pathway of the rat. Exp Brain Res. 62:1986;470-478.
37. Clarke PBS, Hommer DW, Pert A, Skirboll LR. Innervation of substantia nigra neurons by cholinergic afferents from pedunculopontine nucleus in the rat: neuroanatomical and electrophysiological evidence. Neuroscience. 23:1987;1011-1019.
38. Smith AD, Bolam JP. The neural network of the basal ganglia as revealed by the study of synaptic connections of identified neurones. Trends Neurosci. 13:1990;259-265.
39. Berendse HW, Groenewegen HJ, Lohman AHM. Compartmental distribution of ventral striatal neurons projecting to the mesencephalon in the rat. J Neurosci. 12:1992;2079-2103.
40. of outstanding interest Haber S, Kunishio K, Mizobuchi M, Lynd-Balta E. The orbital and medial prefrontal circuit through the primate basal ganglia. J Neurosci. 15:1995;4851-4867 This report, from extensive studies of the anatomical connections of the primate basal ganglia, provides detailed information on the topography of striatonigral projections, which are part of the projections from orbitofrontal cortex via the striatum to the globus pallidus and substantia nigra.
41. Grace AA, Bunney BS. Oppopsing effets of striatonigral feedback pathways on midbrain dopamine cell activity. Brain Res. 333:1985;271-284.
42. Smith ID, Grace AA. Role of the subthalamic nucleus in the regulation of nigral dopamine neuron activity. Synapse. 12:1992;287-303.
43. A receptor-mediated inhibition of rat substantia nigra of dopaminergic neurons by pars reticulata projection neurons. J Neurosci. 15:1995;3092-3103 This study addressed the issue of how activation of the inhibitory striatal projection to substantia nigra [38,39,40] could lead to activation of dopamine neurons. The results suggest that dopamine neurons are released from tonic inhibition of local axon collaterals of reticulata output neurons, in contrast to previous hypotheses assuming inhibitory reticulata interneurons in substantia nigra.
44. Gerfen CR. The neostriatal mosaic: compartmentalization of corticostriatal input and striatonigral output systems. Nature. 311:1984;461-464.
45. Jimenez-Castellanos J, Graybiel AM. Evidence that histochemically distinct zones of the primate substantia nigra pars compacta are related to patterned distributions of nigrostriatal projection neurons and striatonigral fibers. Exp Brain Res. 74:1989;227-238.
46. Sutton RS, Barto AG. Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev. 88:1981;135-170.
47. Sutton RS, Bato AG. Time-derivative models of Pavlovian reinforcement. Gabriel M, Moore J. Learning and Computational Neuroscience: Foundations of Adaptive Networks. 1990;497-537 MIT Press, Cambridge, MA.
48. Montague PR, Dayan P, Nowlan SJ, Pouget A, Sejnowski TJ. Using aperioidic reinforcement for directed self-organization during development. Hanson SJ, Cowan JD, Giles CL. Neural Information Processing Systems 5. 1993;969-976 Morgan Kaufmann, San Mateo, CA.
49. of outstanding interest Houk JC, Adams JL, Barto AG. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Houk JC, Davis JL, Beiser DG. Models of Information Processing in the Basal Ganglia. 1995;249-270 MIT Press, Cambridge, MA, This report together with a previous paper [48] are the first to describe the striking similarities between the teaching signal emitted by the adaptive critic module of TD models [46] and the dopamine neuron response to primary rewards and to conditioned, reward-predicting stimuli. The report, furthermore, describes in detail the possible molecular basis for hypothetical striatal eligibility traces that would be required for modifying the weights of synapses backwards in time that were activated in relation to the stimuli and actions leading up to the reinforcer [46]. The book, as a whole, contains a collection of synthetic descriptions of the structural organization and function of the basal ganglia and provides much helpful information and formal foundations for developing basal ganglia models.
50. of outstanding interest Barto AG. Adaptive critics and the basal ganglia. Houk JC, Davis JL, Beiser DG. Models of Information Processing in the Basal Ganglia. 1995;215-232 MIT Press, Cambridge, MA, A parallel report to the previous paper [49], this text provides an excellent introduction to TD predictive reinforcement models by one of their principal creators. The report also presents an architecture of TD models (see [46]) that closely resembles the anatomical organization of the basal ganglia [38]. With the information provided, it is possible to have a basic TD model with dopamine-like teaching signals up and running within a few days and to start experimenting with it and modifying it using commercial mathematical software (see R Suri, W Schultz, Soc Neurosci Abstr 1996, 22:1389).
51. Barto AG, Sutton RS, Anderson CW. Neuron-like adaptive elements that can solve difficult learning problems. IEEE Trans Syst Man Cybern. 13:1983;834-846.
52. Tesauro G. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural Computation. 6:1994;215-219.
53. Fagg AH. Reinforcement learning for robotic reaching and grasping. Bennet KMB, Castiello U. New Perspectives in the Control of the Reach to Grasp Movement. 1993;281-308 North Holland, Amsterdam.
54. Friston KJ, Tononi G, Reeke GN Jr, Sporns O, Edelman GM. Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience. 59:1994;229-243.
55. of outstanding interest Montague PR, Dayan P, Sejnowski TJ. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci. 16:1996;1936-1947 This is the first explicit and detailed computer model of dopamine neuron responses as teaching signals of the adaptive critic module of a TD predictive reinforcement model. It is applied to human decision-making.
56. of outstanding interest Montague PR, Dayan P, Person C, Sejnowski TJ. Bee foraging in uncertain environments using predictive Hebbian learning. of special interest Nature. 377:1995;725-728 A slightly different version of TD reinforcement model from the one used by Montague et al. [55], based on measured responses of an octopamine neuron in honeybees [57]. This well developed and well presented model was employed to simulate the foraging behavior of bees.
57. Hammer M. An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature. 366:1993;59-63.
58. Gonon FG. Nonlinear relationship between impulse flow and dopamine released by rat midbrain dopaminergic neurons as studied by in vivo electrochemistry. Neuroscience. 24:1988;19-28.
59. Kawagoe KT, Garris PA, Wiedemann DJ, Wightman RM. Regulation of transient dopamine conentration gradients in the microenvironment surrounding nerve terminals in the rat striatum. Neuroscience. 51:1992;55-64.
60. Cepeda C, Buchwald N, Levine MS. Neuromodulatory actions of dopamine in the neostriatum are dependent upon the excitatory amino acid receptor subtype activated. Proc Natl Acad Sci USA. 90:1993;9576-9580.
61. of special interest Gonon F, Sundstrom L. Excitatory effects of dopamine released by impulse flow in the rat nucleus accumbens in vivo. Neuroscience. 75:1996;13-18 This report and a previously published paper [60] are prominent examples on how the synaptic action of dopamine, separated according to an action via D1- and D2-type receptors, might be resolved after more than 20 years of inconsistent results.
62. Colabresi P, Maj R, Pisani A, Mercuri NB, Bernardi G. Long-term synaptic depression in the striatum: physiological and pharmacological characterization. J Neurosci. 12:1992;4224-4233.
63. of outstanding interest Wickens JR, Begg AJ, Arbuthnott GW. Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex in vitro. Neuroscience. 70:1996;1-5 The first report on longer-lasting, dopamine-mediated increases of cortico-striatal transmission. Instead of using bath application of dopamine on striatal slices, this study employed pulsatile dopamine application in order to simulate phasic release from impulse responses. Dopamine was applied simultaneously with corticostriatal tetanic stimulation. In view of the theoretically interesting eligibility traces [46,49], it would be interesting to test whether dopamine application would also be efficient if applied after corticostriatal stimulation. Furthermore, one could test dopamine specificity by blocking the observed effects with dopamine receptor antagonists.
64. Wickens J, Kötter R. Cellular models of reinforcement. Houk JC, Davis JL, Beiser DG. Models of Information Processing in the Basal Ganglia. 1995;187-214 MIT Press, Cambridge, MA.
65. Watanabe M. The appropriateness of behavioral responses coded in post-trial activity of primate prefrontal units. Neurosci Lett. 101:1989;113-117.
66. Thorpe SJ, Rolls ET, Maddison S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp Brain Res. 49:1983;93-115.
67. Niki H, Watanabe M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171:1979;213-224.
68. Hikosaka O, Sakamoto M, Usui S. Functional properties of monkey caudate neurons. III. Activities related to expectation of target and reward. J Neurophysiol. 61:1989;814-832.
69. Apicella P, Ljungberg T, Scarnati E, Schultz W. Responses to reward in monkey dorsal and ventral striatum. Exp Brain Res. 85:1991;491-500.
70. Williams GV, Rolls ET, Leonard CM, Stern C. Neuronal responses in the ventral striatum of the behaving monkey. Behav Brain Res. 55:1993;243-252.
71. Aosaki T, Tsubokawa H, Ishida A, Watanabe K, Graybiel M, Kimura M. Responses of tonically active neurons in the primate's striatum undergo systematic changed during behavioral sensorimotor conditioning. J Neurosci. 14:1994;3969-3984.
72. of outstanding interest Aosaki T, Kimura M, Graybiel AM. Temporal and spatial characteristics of tonically active neurons of the primate's striatum. J Neurophysiol. 73:1995;1234-1252 This and the previous report [71] demonstrate how tonically active striatal neurons acquire depressant responses to visual and auditory reward-predicting stimuli while monkeys undergo classical conditioning.
73. of outstanding interest Apicella P, Legallet E, Trouche E. Responses of tonically discharging neurons in monkey striatum to visual stimuli presented under passive conditions and during task performance. of special interest Neurosci Lett. 203:1996;147-150 Tonically active striatal neurons are phasically depressed by reward-predicting stimuli, independent of a behavioral reaction of the animal. As also judged from previous data [71,72], these responses resemble the phasic activations of dopamine neurons that are known to innervate tonically active striatal neurons.
74. Apicella P, Scarnati E, Ljungberg T, Schultz W. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J Neurophysiol. 68:1992;945-960.
75. Schultz W, Apicella P, Scarnati E, Ljungberg T. Neuronal activity in monkey ventral striatum related to the expectation of reward. J Neurosci. 12:1992;4595-4610.
76. Matsumura M, Kojima J, Gardiner TW, Hikosaka O. Visual and oculomotor functions of monkey subthalamic nucleus. J Neurophysiol. 67:1992;1615-1632.
77. of special interest Rolls ET, Critchley HD, Mason R, Wakeman EA. Orbitofrontal cortex neurons: role in olfactory and visual association learning. J Neurophysiol. 75:1996;1970-1981 Orbitofrontal neurons respond to visual cues dependent on the cue's association with reward. In a visual discrimination task, these neurons differentiate between cues signalling sweet liquid and those signaling aversive saline. Reversal of the cues' associations results in rapid reversal of differential neuronal responses simultaneously with behavioral reversal.
78. Nishijo H, Ono T, Nishino H. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J Neurosci. 8:1988;3570-3583.
79. Watanabe M. Prefrontal unit activity during associative learning in the monkey. Exp Brain Res. 80:1990;296-309.
80. of outstanding interest Watanabe M. Reward expectancy in primate prefrontal neurons. Nature. 382:1996;629-632 This report describes how the typical delay activity of neurons in dorsolateral prefrontal cortex depends on the particular liquid or food reward obtained at trial end in standard spatial delayed response tasks. The results are not entirely unexpected because previous experiments by the author [79] have shown that neuronal responses in prefrontal neurons can depend on stimulus associations with reward as opposed to no reward. As rewards represent goals of action [16], these data correspond well to general notions about prefrontal functions in the control of goal-directed behavior, as derived from human clinical cases and primate lesion experiments. These results may well lead to an enlargement of, or even a shift in paradigm in, neurophysiological concepts of prefrontal functions that presently center around short-term or working memory.
81. Chang JY, Sawyer SF, Lee RS, Woodward DJ. Electrophysiological and pharmacological evidence for the role of the nucleus accumbens in cocaine self-administration in freely moving rats. J Neurosci. 14:1994;1224-1244.
82. Carelli RM, Deadwyler SA. A comparison of nucleus accumbens neuronal firing patterns during cocaine self-administration and water reinforcement in rats. J Neurosci. 14:1994;7735-7746.
83. of outstanding interest Bowman EM, Aigner TG, Richmond BJ. Neural signals in the monkey ventral striatum related to motivation for juice and cocaine rewards. of special interest J Neurophysiol. 75:1996;1061-1073 This study on monkeys and the previous studies on rats [81,82] are the first to use cocaine as a reward while investigating the electrical activity of neurons in brain structures that are known to be involved in drug dependence [8-10,11,12]. Activation occurs in some neurons that are also activated by liquid rewards, although other neurons are exclusively activated by cocaine. Rat and monkey studies have shown that activations occur in relation to movements made for obtaining cocaine, during the expectation of cocaine, and after the delivery of cocaine; these data suggest that the cocaine cue is processed by some neurons as if it were a reward. This suggests that cocaine effects on behavior may, in part, occur through natural reward mechanisms.