A causal link between prediction errors, dopamine neurons and learning

Nature Neuroscience

Volumn 16, Issue 7, 2013, Pages 966-973

  Steinberg, Elizabeth E ^a,b   Keiflin, Ronald ^a   Boivin, Josiah R ^a,b   Witten, Ilana B ^c   Deisseroth, Karl ^d,e   Janak, Patricia H ^a,b,f  

^a UNIVERSITY OF CALIFORNIA SAN FRANCISCO   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

^c PRINCETON UNIVERSITY   (United States)

^d Stanford University   (United States)

^e STANFORD UNIVERSITY   (United States)

^f UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE; BACTERIAL PROTEIN; CHANNELRHODOPSIN 2, MOUSE; PHOTOPROTEIN; RHODOPSIN; TYROSINE 3 MONOOXYGENASE; YELLOW FLUORESCENT PROTEIN, BACTERIA;

ANIMAL EXPERIMENT; ARTICLE; ASSOCIATION; CONCEPTUAL FRAMEWORK; CONTROLLED STUDY; DOPAMINERGIC NERVE CELL; FEMALE; LEARNING; NONHUMAN; PREDICTION; PRIORITY JOURNAL; RAT; REWARD; STATE DEPENDENT LEARNING; ANIMAL; AUDITORY STIMULATION; CYTOLOGY; ELECTROSTIMULATION; ESTRUS CYCLE; FOOD PREFERENCE; GENETICS; HEMISPHERIC DOMINANCE; INSTRUMENTAL CONDITIONING; LONG EVANS RAT; METABOLISM; OPTOGENETICS; PHYSIOLOGY; PREDICTIVE VALUE; REINFORCEMENT; TRANSGENIC RAT; VENTRAL TEGMENTUM;

ACOUSTIC STIMULATION; ANIMALS; ASSOCIATION LEARNING; BACTERIAL PROTEINS; CONDITIONING, OPERANT; CUES; DOPAMINERGIC NEURONS; ELECTRIC STIMULATION; ESTROUS CYCLE; EXTINCTION, PSYCHOLOGICAL; FEMALE; FOOD PREFERENCES; FUNCTIONAL LATERALITY; LUMINESCENT PROTEINS; OPTOGENETICS; PREDICTIVE VALUE OF TESTS; RATS; RATS, LONG-EVANS; RATS, TRANSGENIC; REWARD; RHODOPSIN; TYROSINE 3-MONOOXYGENASE; VENTRAL TEGMENTAL AREA;

EID: 84891681994     PISSN: 10976256     EISSN: 15461726     Source Type: Journal    
DOI: 10.1038/nn.3413     Document Type: Article

References (48)

1. Rescorla, R.A. & Wagner, A.R. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64-99 (Appleton Century Crofts, New York, 1972).
2. Glimcher, P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. USA 108 (suppl. 3): 15647-15654 (2011).
3. Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947 (1996).
4. Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593-1599 (1997).
5. Schultz, W. & Dickinson, A. Neuronal coding of prediction errors. Annu. Rev. Neurosci. 23, 473-500 (2000).
6. Sutton, R.S. & Barto, A.G. Toward a modern theory of adaptive networks: expectation and prediction. Psychol. Rev. 88, 135-170 (1981).
7. Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900-913 (1993).
8. Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B. & Uchida, N. Neuron type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85-88 (2012).
9. Roesch, M.R., Calu, D.J. & Schoenbaum, G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10, 1615-1624 (2007).
10. Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304-309 (1998).
11. Mirenowicz, J. & Schultz, W. Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72, 1024-1027 (1994).
12. Day, J.J., Roitman, M.F., Wightman, R.M. & Carelli, R.M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020-1028 (2007).
13. Takahashi, Y.K. et al. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62, 269-280 (2009).
14. Iordanova, M.D., Westbrook, R.F. & Killcross, A.S. Dopamine activity in the nucleus accumbens modulates blocking in fear conditioning. Eur. J. Neurosci. 24, 3265-3270 (2006).
15. O'Tuathaigh, C.M. et al. The effect of amphetamine on Kamin blocking and overshadowing. Behav. Pharmacol. 14, 315-322 (2003).
16. Parker, J.G. et al. Absence of NMDA receptors in dopamine neurons attenuates dopamine release, but not conditioned approach, during Pavlovian conditioning. Proc. Natl. Acad. Sci. USA 107, 13491-13496 (2010).
17. Zweifel, L.S. et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl. Acad. Sci. USA 106, 7281-7288 (2009).
18. Adamantidis, A.R. et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829-10835 (2011).
19. Domingos, A.I. et al. Leptin regulates the reward value of nutrient. Nat. Neurosci. 14, 1562-1568 (2011).
20. Tsai, H.C. et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080-1084 (2009).
21. Witten, I.B. et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721-733 (2011).
22. Boyden, E.S., Zhang, F., Bamberg, E., Nagel, G. & Deisseroth, K. Millisecond-timescale, genetically targeted optical control of neural activity. Nat. Neurosci. 8, 1263-1268 (2005).
23. Zhang, F., Wang, L.P., Boyden, E.S. & Deisseroth, K. Channelrhodopsin-2 and optical control of excitable cells. Nat. Methods 3, 785-792 (2006).
24. Kamin, L.J. "Attention-like" processes in classical conditioning. in Miami Symposium on the Prediction of Behavior, 1967: Aversive Stimulation (ed. Jones, M.R.) 9-31 (University of Miami Press, 1968).
25. Kamin, L.J. Selective association and conditioning. in Fundamental Issues in Associative Learning (eds. Mackintosh, N.J. & Honig, F.W.K.) 42-64 (Dalhousie University Press, 1969).
26. Kamin, L.J. Predictability, surprise, attention and conditioning. in Punishment and Aversive Behavior (eds. Campbell, B.A. & Church, R.M.) 279-296 (Appleton-Century-Crofts, New York, NY, 1969).
27. Holland, P.C. Unblocking in Pavlovian appetitive conditioning. J. Exp. Psychol. Anim. Behav. Process. 10, 476-497 (1984).
28. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43-48 (2001).
29. Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1-27 (1998).
30. Burke, K.A., Franz, T.M., Miller, D.N. & Schoenbaum, G. The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards. Nature 454, 340-344 (2008).
31. Daw, N.D., Kakade, S. & Dayan, P. Opponent interactions between serotonin and dopamine. Neural Netw. 15, 603-616 (2002).
32. Peters, J., Kalivas, P.W. & Quirk, G.J. Extinction circuits for fear and addiction overlap in prefrontal cortex. Learn. Mem. 16, 279-288 (2009).
33. Becker, J.B. Gender differences in dopaminergic function in striatum and nucleus accumbens. Pharmacol. Biochem. Behav. 64, 803-812 (1999).
34. Berridge, K.C. & Robinson, T.E. What is the role of dopamine in reward: hedonic impact, reward learning or incentive salience? Brain Res. Rev. 28, 309-369 (1998).
35. Wassum, K.M., Ostlund, S.B., Balleine, B.W. & Maidment, N.T. Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling. Learn. Mem. 18, 475-483 (2011).
36. Beckstead, R.M., Domesick, V.B. & Nauta, W.J. Efferent connections of the substantia nigra and ventral tegmental area in the rat. Brain Res. 175, 191-217 (1979).
37. Fields, H.L., Hjelmstad, G.O., Margolis, E.B. & Nicola, S.M. Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu. Rev. Neurosci. 30, 289-316 (2007).
38. Swanson, L.W. The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res. Bull. 9, 321-353 (1982).
39. Reynolds, J.N. & Wickens, J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15, 507-521 (2002).
40. Wickens, J.R., Horvitz, J.C., Costa, R.M. & Killcross, S. Dopaminergic mechanisms in actions and habits. J. Neurosci. 27, 8181-8183 (2007).
41. Gerfen, C.R. & Surmeier, D.J. Modulation of striatal projection systems by dopamine. Annu. Rev. Neurosci. 34, 441-466 (2011).
42. Reynolds, J.N., Hyland, B.I. & Wickens, J.R. A cellular mechanism of reward-related learning. Nature 413, 67-70 (2001).
43. Tye, K.M. et al. Methylphenidate facilitates learning-induced amygdala plasticity. Nat. Neurosci. 13, 475-481 (2010).
44. Stuber, G.D. et al. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science 321, 1690-1692 (2008).
45. Brown, M.T. et al. Drug-driven AMPA receptor redistribution mimicked by selective dopamine neuron stimulation. PLoS ONE 5, e15870 (2010).
46. Suri, R.E. TD models of reward predictive responses in dopamine neurons. Neural Netw. 15, 523-533 (2002).
47. Flagel, S.B. et al. A selective role for dopamine in stimulus-reward learning. Nature 469, 53-57 (2011).
48. Karim, B.O. et al. Estrous cycle and ovarian changes in a rat mammary carcinogenesis model after irradiation, tamoxifen chemoprevention and aging. Comp. Med. 53, 532-538 (2003).