The short-latency dopamine signal: A role in discovering novel actions?

Nature Reviews Neuroscience

Volumn 7, Issue 12, 2006, Pages 967-975

  Redgrave, Peter ^a   Gurney, Kevin ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

BASAL GANGLION; DOPAMINERGIC ACTIVITY; LEARNING; NERVE CELL; NOCICEPTIVE STIMULATION; NONHUMAN; PRIORITY JOURNAL; REINFORCEMENT; REVIEW; SENSORY NERVE CELL; SIGNAL PROCESSING; VISION;

ANIMALS; BRAIN; DOPAMINE; EXPLORATORY BEHAVIOR; HUMANS; MODELS, NEUROLOGICAL; NEURAL PATHWAYS; REACTION TIME; REINFORCEMENT (PSYCHOLOGY); REWARD; SYNAPTIC TRANSMISSION;

EID: 33751184634     PISSN: 1471003X     EISSN: 14710048     Source Type: Journal    
DOI: 10.1038/nrn2022     Document Type: Review

References (125)

1. Thorndike, E. L. Animal Intelligence (Macmillan, New York, 1911).
2. Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27 (1998).
3. Redgrave, P., Prescott, T. J. & Gurney, K. Is the short latency dopamine response too short to signal reward error? Trends Neurosci. 22, 146-151 (1999).
4. Comoli, E. et al. A direct projection from superior colliculus to substantia nigra for detecting salient visual events. Nature Neurosci. 6, 974-980 (2003).
5. Dommett, E. et al. How visual stimuli activate dopaminergic neurons at short latency. Science 307, 1476-1479 (2005).
6. Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947 (1996).
7. Montague, P. R., Hyman, S. E. & Cohen, J. D. Computational roles for dopamine in behavioural control. Nature 431, 760-767 (2004).
8. Schultz, W. Getting formal with dopamine and reward. Neuron 36, 241-263 (2002).
9. Schultz, W. Behavioral theories and the neurophysiology of reward. Annu. Rev. Psychol. 57, 87-115 (2006).
10. Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473-500 (2000).
11. Gerfen, C. R. & Wilson, C. J. in Handbook of Chemical Neuroanatomy Vol. 12 (eds Swanson, L. W., Bjorklund, A. & Hokfelt, T.) Part III, 371-468 (Elsevier, Amsterdam, 1996).
12. Graybiel, A. M. Neurotransmitter and neuromodulators in the basal ganglia. Trends Neurosci. 13, 244-254 (1990).
13. Hiroi, N. et al. Molecular dissection of dopamine receptor signaling. J. Chem. Neuroanat. 23, 237-242 (2002).
14. Bergman, H. et al. Physiological aspects of information processing in the basal ganglia of normal and Parkinsonian primates. Trends Neurosci. 21, 32-38 (1998).
15. Radad, K., Gille, G. & Rausch, W. D. Short review on dopamine agonists: insight into clinical and research studies relevant to Parkinson's disease. Pharm. Rep. 57, 701-712 (2005).
16. Wise, R. A. Dopamine, learning and motivation. Nature Rev. Neurosci. 5, 483-494 (2004).
17. Berridge, K. C. & Robinson, T. E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res. Rev. 28, 309-369 (1998).
18. Salamone, J. D. & Correa, M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav. Brain Res. 137, 3-25 (2002).
19. Marr, D. Vision: A Computational Approach (Freeman & Co., San Francisco, 1982).
20. Gurney, K., Prescott, T. J., Wickens, J. R. & Redgrave, P. Computational models of the basal ganglia: from robots to membranes. Trends Neurosci. 27, 453-459 (2004).
21. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43-48 (2001).
22. Fiorillo, C. D., Tobler, P. N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898-1902 (2003).
23. Tobler, P. N., Fiorillo, C. D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642-1645 (2005).
24. Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129-141 (2005).
25. Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913-9923 (2003).
26. Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269-280 (2004).
27. Tobler, P. N., Dickinson, A. & Schultz, W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 23, 10402-10410 (2003).
28. Ungless, M. A. Dopamine: the salient issue. Trends Neurosci. 27, 702-706 (2004).
29. Sugrue, L. P., Corrado, G. S. & Newsome, W. T. Choosing the greater of two goods: neural currencies for valuation and decision making. Nature Rev. Neurosci. 6, 363-375 (2005).
30. Salzman, C. D., Belova, M. A. & Paton, J. J. Beetles, boxes and brain cells: neural mechanisms underlying valuation and learning. Curr. Opin. Neurobiol. 15, 721-729 (2005).
31. Houk, J. C. Agents of the mind. Biol. Cybern. 92, 427-437 (2005).
32. Suri, R. E. TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523-533 (2002).
33. Bar-Gad, I. & Bergman, H. Stepping out of the box: information processing in the neural networks of the basal ganglia. Curr. Opin. Neurobiol. 11, 689-695 (2001).
34. Frank, M. J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 17, 51-72 (2005).
35. Daw, N. D., Niv, Y. & Dayan, P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neurosci. 8, 1704-1711 (2005).
36. Freeman, A. S. Firing properties of substantia nigra dopaminergic neurons in freely moving rats. Life Sci. 36, 1983-1994 (1985).
37. Guarraci, F. A. & Kapp, B. S. An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential pavlovian fear conditioning in the awake rabbit. Behav. Brain Res. 99, 169-179 (1999).
38. Horvitz, J. C., Stewart, T. & Jacobs, B. L. Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res. 759, 251-258 (1997).
39. Ljungberg, T., Apicella, P. & Schultz, W. Responses of monkey dopamine neurons during learning of behavioural reactions. J. Neurophysiol. 67, 145-163 (1992).
40. Pan, W. X., Schmidt, R., Wickens, J. R. & Hyland, B. I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward - learning network. J. Neurosci. 25, 6235-6242 (2005).
41. Schultz, W., Dayan, P. & Montague, P. R. A neural substrate of prediction and reward. Science 275, 1593-1599 (1997).
42. Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024-1027 (1994).
43. Coizet, V., Comoli, E., Westby, G. W. M. & Redgrave, P. Phasic activation of substantia nigra and the ventral tegmental area by chemical stimulation of the superior colliculus: an electrophysiological investigation in the rat. Eur. J. Neurosci. 17, 28-40 (2003).
44. Overton, P. G., Coizet, V., Dommett, E. J. & Redgrave, P. The parabrachial nucleus is a source of short latency nociceptive input to midbrain dopaminergic neurones in rat. Soc. Neurosci. Abstr. 301.5 (2005).
45. Coizet, V., Dommett, E. J., Redgrave, P. & Overton, P. G. Nociceptive responses of midbrain dopaminergic neurones are modulated by the superior colliculus in the rat. Neuroscience 139, 1479-1493 (2006).
46. McHaffie, J. G. et al. A direct projection from superior colliculus to substantia nigra pars compacta in the cat. Neuroscience 138, 221-234 (2006).
47. Horvitz, J. C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651-656 (2000).
48. Takikawa, Y., Kawagoe, R. & Hikosaka, O. A possible role of midbrain dopamine neurons in short - and long-term adaptation of saccades to position-reward mapping. J. Neurophysiol. 92, 2520-2529 (2004).
49. Jay, M. F. & Sparks, D. L. Sensorimotor integration in the primate superior colliculus. I. Motor convergence. J. Neurophysiol. 57, 22-34 (1987).
50. Hikosaka, O. & Wurtz, R. H. Visual and oculomotor function of monkey substantia nigra pars reticulata. I. Relation of visual and auditory responses to saccades. J. Neurophysiol. 49, 1230-1253 (1983).
51. Thorpe, S. J. & Fabre-Thorpe, M. Seeking categories in the brain. Science 291, 260-263 (2001).
52. Rousselet, G. A., Thorpe, S. J. & Fabre-Thorpe, M. How parallel is visual processing in the ventral pathway? Trends Cogn. Sci. 8, 363-370 (2004).
53. Schultz, W. & Romo, R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioural reactions. J. Neurophysiol. 63, 607-624 (1990).
54. Hikosaka, O., Sakamoto, M. & Usui, S. Functional properties of monkey caudate neurons. II. Visual and auditory responses. J. Neurophysiol. 61, 799-813 (1989).
55. Matsumura, M., Kojima, J., Gardiner, T. W. & Hikosaka, O. Visual and oculomotor functions of monkey subthalamic nucleus. J. Neurophysiol. 67, 1615-1632 (1992).
56. May, P. J. et al. Projections from the superior colliculus to substantia nigra pars compacta in a primate. Soc. Neurosci. Abstr. 450.2 (2005).
57. A receptor-mediated inhibition unmasks interlaminar connection within superior colliculus in anesthetized adult rats. Neurosci. Res. 46, 73-83 (2003).
58. Wurtz, R. H. & Albano, J. E. Visual-motor function of the primate superior colliculus. Ann. Rev. Neurosci. 3, 189-226 (1980).
59. Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of the primate superior colliculus. Physiol. Rev. 66, 118-171 (1986).
60. Grantyn, R. in Neuroanatomy of the Oculomotor System (ed. Buttner-Ennever, J. A.) 273-333 (Elsevier, Amsterdam, 1988).
61. Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, Cambridge, Massachusetts, 1993).
62. Horn, G. & Hill, R. M. Effect of removing the neocortex on the response to repeated sensory stimulation of neurones in the mid-brain. Nature 211, 754-755 (1966).
63. Sprague, J. M., Marchiafava, P. L. & Rixxolatti, G. Unit responses to visual stimuli in the superior colliculus of the unanesthetized, mid-pontine cat. Arch. Ital. Biol. 106, 169-193 (1968).
64. Ikeda, T. & Hikosaka, O. Reward-dependent gain and bias of visual responses in primate superior colliculus. Neuron 39, 693-700 (2003).
65. Hikosaka, O., Nakamura, K. & Nakahara, H. Basal ganglia orient eyes to reward. J. Neurophysiol. 95, 567-584 (2006).
66. Sutton, R. S. & Barto, A. G. Reinforcement Learning - an Introduction (MIT Press, Cambridge, Massachusetts, 1998).
67. White, N. M. Reward or reinforcement: what's the difference? Neurosci. Biobehav. Rev. 13, 181-186 (1989).
68. McHaffie, J. G., Stanford, T. R., Stein, B. E., Coizet, V. & Redgrave, P. Subcortical loops through the basal ganglia. Trends Neurosci. 28, 401-407 (2005).
69. Reynolds, J. N. J., Schulz, J. M. & Wickens, J. R. Visual responsiveness of striatal spiny neurons in anaesthetised rats: an in vivo intracellular study. Proc. Int. Australas. Wint. Conf. Brain Res. Abstr. 6.4, 39 (2005).
70. Schultz, W., Apicella, P., Romo, R. & Scarnati, E. in Models of Information Processing in the Basal Ganglia (eds Houk, J. C., Davis, J. L. & Beiser, D. G.) 11-27 (MIT Press, Cambridge, Massachusetts, 1995).
71. Apicella, P., Legallet, E. & Trouche, E. Responses of tonically discharging neurons in the monkey striatum to primary rewards delivered during different behavioral states. Exp. Brain Res. 116, 456-466 (1997).
72. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337-1340 (2005).
73. Crutcher, M. D. & DeLong, M. R. Single cell studies of the primate putamen. II. Relations to direction of movement and pattern of muscular activity. Exp. Brain Res. 53, 244-258 (1984).
74. Bickford, M. E. & Hall, W. C. Collateral projections of predorsal bundle cells of the superior colliculus in the rat. J. Comp. Neurol. 283, 86-106 (1989).
75. Levesque, M., Charara, A., Gagnon, S., Parent, A. & Deschenes, M. Corticostriatal projections from layer V cells in rat are collaterals of long-range corticofugal axons. Brain Res. 709, 311-315 (1996).
76. Mink, J. W. The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50, 381-425 (1996).
77. Reiner, A., Jiao, Y., DelMar, N., Laverghetta, A. V. & Lei, W. L. Differential morphology of pyramidal tract-type and intratelencephalically projecting-type corticostriatal neurons and their intrastriatal terminals in rats. J. Comp. Neurol. 457, 420-440 (2003).
78. Alexander, G. E., DeLong, M. R. & Strick, P. L. Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Ann. Rev. Neurosci. 9, 357-381 (1986).
79. Haber, S. N. The primate basal ganglia: parallel and integrative networks. J. Chem. Neuroanat. 26, 317-330 (2003).
80. Harting, J. K., Updyke, B. V. & VanLieshout, D. P. The visual-oculomotor striatum of the cat: functional relationship to the superior colliculus. Exp. Brain Res. 136, 138-142 (2001).
81. Krout, K. E., Loewy, A. D., Westby, G. W. M. & Redgrave, P. Superior colliculus projections to midline and intralaminar thalamic nuclei of the rat. J. Comp. Neurol. 431, 198-216 (2001).
82. Krout, K. E., Belzer, R. E. & Loewy, A. D. Brainstem projections to midline and intralaminar thalamic nuclei of the rat. J. Comp. Neurol. 448, 53-101 (2002).
83. Van der Werf, Y. D., Witter, M. P. & Groenewegen, H. J. The intralaminar and midline nuclei of the thalamus. Anatomical and functional evidence for participation in processes of arousal and awareness. Brain Res. Rev. 39, 107-140 (2002).
84. Smith, Y., Raju, D. V., Pare, J. F. & Sidibe, M. The thalamostriatal system: a highly specific network of the basal ganglia circuitry. Trends Neurosci. 27, 520-527 (2004).
85. Parent, M. & Parent, A. Single-axon tracing and three-dimensional reconstruction of centre median-parafascicular thalamic neurons in primates. J. Comp. Neurol. 481, 127-144 (2005).
86. Matsumoto, N., Minamimoto, T., Graybiel, A. M. & Kimura, M. Neurons in the thalamic CM-Pf complex supply striatal neurons with information about behaviorally significant sensory events. J. Neurophysiol. 85, 960-976 (2001).
87. Wightman, R. M. & Robinson, D. L. Transient changes in mesolimbic dopamine and their association with 'reward'. J. Neurochem. 82, 721-735 (2002).
88. Roitman, M. F., Stuber, G. D., Phillips, P. E. M., Wightman, R. M. & Carelli, R. M. Dopamine operates as a subsecond modulator of food seeking. J. Neurosci. 24, 1265-1271 (2004).
89. Centonze, D., Picconi, B., Gubellini, P., Bernardi, G. & Calabresi, P. Dopaminergic control of synaptic plasticity in the dorsal striatum. Eur. J. Neurosci. 13, 1071-1077 (2001).
90. Reynolds, J. N. & Wickens, J. R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507-521 (2002).
91. Wickens, J. A Theory of the Striatum (Pergamon, Oxford, 1993).
92. Hikosaka, O. in The Basal ganglia IV: New Ideas and Data on Structure and Function (eds Percheron, G., McKenzie, J. S. & Feger, J.) 589-596 (Plenum, New York, 1994).
93. Redgrave, P., Prescott, T. & Gurney, K. N. The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89, 1009-1023 (1999).
94. Gurney, K., Prescott, T. J. & Redgrave, P. A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biol. Cybern. 84, 401-410 (2001).
95. Gurney, K., Prescott, T. J. & Redgrave, P. A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biol. Cybern. 84, 411-423 (2001).
96. Prescott, T. J., Gonzalez, F. M. M., Gurney, K., Humphries, M. D. & Redgrave, P. A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw. 19, 31-61 (2006).
97. Devenport, L. D. & Holloway, F. A. The rat's resistance to superstition: role of the hippocampus. J. Comp. Physiol. Psychol. 94, 691-705 (1980).
98. Roberts, S. & Gharib, A. Variation of bar-press duration: where do new responses come from? Behav. Processes 72, 215-223 (2006).
99. Wickens, J. R., Reynolds, J. N. J. & Hyland, B. I. Neural mechanisms of reward-related motor learning. Curr. Opin. Neurobiol. 13, 685-690 (2003).
100. Paton, J. J., Belova, M. A., Morrison, S. E. & Salzman, C. D. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439, 865-870 (2006).
101. Lisman, J. E. & Grace, A. A. The hippocampal-VTA loop: controlling the entry of information into long-term memory. Neuron 46, 703-713 (2005).
102. Schultz, W. Multiple reward signals in the brain. Nature Rev. Neurosci. 1, 199-207 (2000).
103. Schoenbaum, G., Setlow, B., Saddoris, M. P. & Gallagher, M. Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 39, 855-867 (2003).
104. Corbit, L. H., Ostlund, S. B. & Balleine, B. W. Sensitivity to instrumental contingency degradation is mediated by the entorhinal cortex and its efferents via the dorsal hippocampus. J. Neurosci. 22, 10976-10984 (2002).
105. Corbit, L. H. & Balleine, B. W. The role of prelimbic cortex in instrumental conditioning. Behav. Brain Res. 146, 145-157 (2003).
106. Padoa-Schioppa, C. & Assad, J. A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223-226 (2006).
107. Ungless, M. A., Magill, P. J. & Bolam, J. P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303, 2040-2042 (2004).
108. Klop, E. M., Mouton, L. J., Hulsebosch, R., Boers, J. & Holstege, G. In cat four times as many lamina I neurons project to the parabrachial nuclei and twice as many to the periaqueductal gray as to the thalamus. Neuroscience 134, 189-197 (2005).
109. Dean, P., Redgrave, P. & Westby, G. W. M. Event or emergency? Two response systems in the mammalian superior colliculus. Trends Neurosci. 12, 137-147 (1989).
110. Dickinson, A. The 28th Bartlett Memorial Lecture. Causal learning: an associative analysis. Q. J. Exp. Psychol. B 54, 3-25 (2001).
111. Elsner, B. & Hommel, B. Contiguity and contingency in action-effect learning. Psychol. Res. 68, 138-154 (2004).
112. Yin, H. H., Knowlton, B. J. & Balleine, B. W. Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning. Eur. J. Neurosci. 22, 505-512 (2005).
113. Burgdorf, J. & Panksepp, J. The neurobiology of positive emotions. Neurosci. Biobehav. Rev. 30, 173-187 (2006).
114. Roesch, M. R. & Olson, C. R. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304, 307-310 (2004).
115. McDonald, A. J. Topographical organization of amygdaloid projections to the caudatoputamen, nucleus accumbens, and related striatal-like areas of the rat brain. Neuroscience 44, 15-33 (1991).
116. Fudge, J. L., Kunishio, K., Walsh, P., Richard, C. & Haber, S. N. Amygdaloid projections to ventromedial striatal subterritories in the primate. Neuroscience 110, 257-275 (2002).
117. Singh, S., Barto, A. G. & Chentanez, N. in Advances in Neural Information Processing Systems 17 (eds Saul, L. K., Weiss, H. & Bottou, L.) 1281-1288 (MIT Press, Cambridge, Massachusetts, 2005).
118. Robbins, T. W. & Sahakian, B. J. in Metabolic Disorders of the Nervous System (ed. Rose, F. C.) 244-291 (Pitman, London, 1981).
119. Saka, E., Goodrich, C., Harlan, P., Madras, B. K. & Graybiel, A. M. Repetitive behaviors in monkeys are linked to specific striatal activation patterns. J. Neurosci. 24, 7557-7565 (2004).
120. Daprati, E. et al. Looking for the agent: an investigation into consciousness of action and self-consciousness in schizophrenic patients. Cognition 65, 71-86 (1997).
121. Spence, S. A. et al. A PET study of voluntary movement in schizophrenic patients experiencing passivity phenomena (delusions of alien control). Brain 120, 1997-2011 (1997).
122. Kapur, S., Mizrahi, R. & Li, M. From dopamine to salience to psychosis - linking biology, pharmacology and phenomenology of psychosis. Schiz. Res. 79, 59-68 (2005).
123. Wise, S. P., Murray, E. A. & Gerfen, C. R. The frontal-cortex-basal ganglia system in primates. Crit. Rev. Neurobiol. 10, 317-356 (1996).
124. Reed, P., Mitchell, C. & Nokes, T. Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-level discrimination task. Anim. Learn. Behav. 24, 38-45 (1996).
125. St Clair-Smith, R. & MacLaren, D. Response preconditioning effects. J. Exp. Psychol Anim. Behav. Process. 9, 41-48 (1983).