Temporal Specificity of Reward Prediction Errors Signaled by Putative Dopamine Neurons in Rat VTA Depends on Ventral Striatum

Neuron

Volumn 91, Issue 1, 2016, Pages 182-193

  Takahashi, Yuji K ^a   Langdon, Angela J ^b   Niv, Yael ^b   Schoenbaum, Geoffrey ^a,c,d  

^a NATIONAL INSTITUTE ON DRUG ABUSE   (United States)

^b PRINCETON UNIVERSITY   (United States)

^c UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE   (United States)

^d JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL EXPERIMENT; ANIMAL MODEL; ARTICLE; CONTROLLED STUDY; DOPAMINERGIC NERVE CELL; MALE; MATHEMATICAL MODEL; NONHUMAN; PREDICTION; PRIORITY JOURNAL; RAT; REWARD; REWARD PREDICTION ERROR; SIGNAL TRANSDUCTION; STIMULUS RESPONSE; TASK PERFORMANCE; VENTRAL STRIATUM; VENTRAL TEGMENTUM; ANIMAL; BASAL GANGLION; LONG EVANS RAT; METABOLISM; PHYSIOLOGY;

DOPAMINE;

ANIMALS; BASAL GANGLIA; DOPAMINE; DOPAMINERGIC NEURONS; RATS, LONG-EVANS; REWARD; VENTRAL STRIATUM; VENTRAL TEGMENTAL AREA;

EID: 84991262300     PISSN: 08966273     EISSN: 10974199     Source Type: Journal    
DOI: 10.1016/j.neuron.2016.05.015     Document Type: Article

References (63)

1. Barto, A.G., Adaptive critics and the basal ganglia. Houk, J.C., Davis, J.L., Beiser, D.G., (eds.) Models of Information Processing in the Basal Ganglia, 1995, MIT Press, 215–232.
2. Berridge, K.C., Robinson, T.E., What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Res. Brain Res. Rev. 28 (1998), 309–369.
3. Bissonette, G.B., Burton, A.C., Gentry, R.N., Goldstein, B.L., Hearn, T.N., Barnett, B.R., Kashtelyan, V., Roesch, M.R., Separate populations of neurons in ventral striatum encode value and motivation. PLoS ONE, 8, 2013, e64673.
4. Bocklisch, C., Pascoli, V., Wong, J.C.Y., House, D.R.C., Yvon, C., de Roo, M., Tan, K.R., Lüscher, C., Cocaine disinhibits dopamine neurons by potentiation of GABA transmission in the ventral tegmental area. Science 341 (2013), 1521–1525.
5. Bromberg-Martin, E.S., Matsumoto, M., Hong, S., Hikosaka, O., A pallidus-habenula-dopamine pathway signals inferred stimulus values. J. Neurophysiol. 104 (2010), 1068–1076.
6. Burton, A.C., Bissonette, G.B., Lichtenberg, N.T., Kashtelyan, V., Roesch, M.R., Ventral striatum lesions enhance stimulus and response encoding in dorsal striatum. Biol. Psychiatry 75 (2014), 132–139.
7. Bush, R.R., Mosteller, F., A mathematical model for simple learning. Psychol. Rev. 58 (1951), 313–323.
8. Cohen, J.Y., Haesler, S., Vong, L., Lowell, B.B., Uchida, N., Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482 (2012), 85–88.
9. Dalton, G.L., Phillips, A.G., Floresco, S.B., Preferential involvement by nucleus accumbens shell in mediating probabilistic learning and reversal shifts. J. Neurosci. 34 (2014), 4618–4626.
10. Daw, N.D., Niv, Y., Dayan, P., Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8 (2005), 1704–1711.
11. Daw, N.D., Courville, A.C., Touretzky, D.S., Representation and timing in theories of the dopamine system. Neural Comput. 18 (2006), 1637–1677.
12. Di Chiara, G., Nucleus accumbens shell and core dopamine: differential role in behavior and addiction. Behav. Brain Res. 137 (2002), 75–114.
13. Eichenbaum, H., Time cells in the hippocampus: a new dimension for mapping memories. Nat. Rev. Neurosci. 15 (2014), 732–744.
14. Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., Uchida, N., Arithmetic and local circuitry underlying dopamine prediction errors. Nature 525 (2015), 243–246.
15. Fiorillo, C.D., Newsome, W.T., Schultz, W., The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci. 11 (2008), 966–973.
16. Floresco, S.B., Todd, C.L., Grace, A.A., Glutamatergic afferents from the hippocampus to the nucleus accumbens regulate activity of ventral tegmental area dopamine neurons. J. Neurosci. 21 (2001), 4915–4922.
17. Grace, A.A., Bunney, B.S., Opposing effects of striatonigral feedback pathways on midbrain dopamine cell activity. Brain Res. 333 (1985), 271–284.
18. Groenewegen, H.J., Berendse, H.W., Wolters, J.G., Lohman, A.H.M., The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: evidence for a parallel organization. Prog. Brain Res. 85 (1990), 95–116, discussion 116–118.
19. Hauber, W., Bohn, I., Giertler, C., NMDA, but not dopamine D(2), receptors in the rat nucleus accumbens areinvolved in guidance of instrumental behavior by stimuli predicting reward magnitude. J. Neurosci. 20 (2000), 6282–6288.
20. Hollerman, J.R., Schultz, W., Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1 (1998), 304–309.
21. Jin, X., Costa, R.M., Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 466 (2010), 457–462.
22. Jo, Y.S., Mizumori, S.J., Prefrontal regulation of neuronal activity in the ventral tegmental area. Cereb. Cortex, 2015, bhv215 epub ahead of print.
23. Jo, Y.S., Lee, J., Mizumori, S.J., Effects of prefrontal cortical inactivation on neural activity in the ventral tegmental area. J. Neurosci. 33 (2013), 8159–8171.
24. Joel, D., Niv, Y., Ruppin, E., Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw. 15 (2002), 535–547.
25. Klein-Flügge, M.C., Hunt, L.T., Bach, D.R., Dolan, R.J., Behrens, T.E.J., Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron 72 (2011), 654–664.
26. Kobayashi, S., Schultz, W., Influence of reward delays on responses of dopamine neurons. J. Neurosci. 28 (2008), 7837–7846.
27. Li, X., Qi, J., Yamaguchi, T., Wang, H.-L., Morales, M., Heterogeneous composition of dopamine neurons of the rat A10 region: molecular evidence for diverse signaling properties. Brain Struct. Funct. 218 (2013), 1159–1176.
28. Margolis, E.B., Lock, H., Hjelmstad, G.O., Fields, H.L., The ventral tegmental area revisited: is there an electrophysiological marker for dopaminergic neurons?. J. Physiol. 577 (2006), 907–924.
29. Matsumoto, M., Hikosaka, O., Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459 (2009), 837–841.
30. McDannald, M.A., Lucantonio, F., Burke, K.A., Niv, Y., Schoenbaum, G., Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 31 (2011), 2700–2705.
31. Meck, W.H., Hippocampal function is required for feedback control of an internal clock's criterion. Behav. Neurosci. 102 (1988), 54–60.
32. Meck, W.H., Penney, T.B., Pouthas, V., Cortico-striatal representation of time in animals and humans. Curr. Opin. Neurobiol. 18 (2008), 145–152.
33. Meck, W.H., Church, R.M., Olton, D.S., Hippocampus, time, and memory. Behav. Neurosci. 127 (2013), 642–654.
34. Mello, G.B., Soares, S., Paton, J.J., A scalable population code for time in the striatum. Curr. Biol. 25 (2015), 1113–1122.
35. Mirenowicz, J., Schultz, W., Importance of unpredictability for reward responses in primate dopamine neurons. J. Neurophysiol. 72 (1994), 1024–1027.
36. Mogenson, G.J., Jones, D.L., Yim, C.Y., From motivation to action: functional interface between the limbic system and the motor system. Prog. Neurobiol. 14 (1980), 69–97.
37. Montague, P.R., Dayan, P., Sejnowski, T.J., A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16 (1996), 1936–1947.
38. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., Bergman, H., Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9 (2006), 1057–1063.
39. Nicola, S.M., The nucleus accumbens as part of a basal ganglia action selection circuit. Psychopharmacology (Berl.) 191 (2007), 521–550.
40. O'Doherty, J., Dayan, P., Friston, K.J., Critchley, H., Dolan, R.J., Temporal difference models and reward-related learning in the human brain. Neuron 38 (2003), 329–337.
41. O'Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., Dolan, R.J., Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304 (2004), 452–454.
42. Oleson, E.B., Beckert, M.V., Morra, J.T., Lansink, C.S., Cachope, R., Abdullah, R.A., Loriaux, A.L., Schetters, D., Pattij, T., Roitman, M.F., et al. Endocannabinoids shape accumbal encoding of cue-motivated behavior via CB1 receptor activation in the ventral tegmentum. Neuron 73 (2012), 360–373.
43. Pan, W.-X., Schmidt, R., Wickens, J.R., Hyland, B.I., Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25 (2005), 6235–6242.
44. Rescorla, R.A., Wagner, A.R., A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. Black, A.H., Prokasy, W.F., (eds.) Classical Conditioning II: Current Research and Theory, 1972, Appleton-Century-Crofts, 64–99.
45. Roesch, M.R., Taylor, A.R., Schoenbaum, G., Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51 (2006), 509–520.
46. Roesch, M.R., Calu, D.J., Schoenbaum, G., Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 10 (2007), 1615–1624.
47. Roesch, M.R., Singh, T., Brown, P.L., Mullins, S.E., Schoenbaum, G., Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. J. Neurosci. 29 (2009), 13365–13376.
48. Schultz, W., Dayan, P., Montague, P.R., A neural substrate of prediction and reward. Science 275 (1997), 1593–1599.
49. Seymour, B., O'Doherty, J.P., Dayan, P., Koltzenburg, M., Jones, A.K., Dolan, R.J., Friston, K.J., Frackowiak, R.S., Temporal difference models describe higher-order learning in humans. Nature 429 (2004), 664–667.
50. Singh, T., McDannald, M.A., Takahashi, Y.K., Haney, R.Z., Cooch, N.K., Lucantonio, F., Schoenbaum, G., The role of the nucleus accumbens in knowing when to respond. Learn. Mem. 18 (2011), 85–87.
51. St Onge, J.R., Stopper, C.M., Zahm, D.S., Floresco, S.B., Separate prefrontal-subcortical circuits mediate different components of risk-based decision making. J. Neurosci. 32 (2012), 2886–2899.
52. Steinberg, E.E., Boivin, J.R., Saunders, B.T., Witten, I.B., Deisseroth, K., Janak, P.H., Positive reinforcement mediated by midbrain dopamine neurons requires D1 and D2 receptor activation in the nucleus accumbens. PLoS ONE, 9, 2014, e94771.
53. Stopper, C.M., Floresco, S.B., Contributions of the nucleus accumbens and its subregions to different aspects of risk-based decision making. Cogn. Affect. Behav. Neurosci. 11 (2011), 97–112.
54. Sutton, R.S., Barto, A.G., Reinforcement Learning: An Introduction. 1998, MIT Press.
55. Takahashi, Y.K., Roesch, M.R., Stalnaker, T.A., Haney, R.Z., Calu, D.J., Taylor, A.R., Burke, K.A., Schoenbaum, G., The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62 (2009), 269–280.
56. Takahashi, Y.K., Roesch, M.R., Wilson, R.C., Toreson, K., O'Donnell, P., Niv, Y., Schoenbaum, G., Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat. Neurosci. 14 (2011), 1590–1597.
57. Tian, J., Uchida, N., Habenula lesions reveal that multiple mechanisms underlie dopamine prediction errors. Neuron 87 (2015), 1304–1316.
58. Ungless, M.A., Grace, A.A., Are you or aren't you? Challenges associated with physiologically identifying dopamine neurons. Trends Neurosci. 35 (2012), 422–430.
59. Voorn, P., Vanderschuren, L.J.M.J., Groenewegen, H.J., Robbins, T.W., Pennartz, C.M.A., Putting a spin on the dorsal-ventral divide of the striatum. Trends Neurosci. 27 (2004), 468–474.
60. Waelti, P., Dickinson, A., Schultz, W., Dopamine responses comply with basic assumptions of formal learning theory. Nature 412 (2001), 43–48.
61. Watabe-Uchida, M., Zhu, L., Ogawa, S.K., Vamanrao, A., Uchida, N., Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron 74 (2012), 858–873.
62. Willuhn, I., Burgeno, L.M., Everitt, B.J., Phillips, P.E.M., Hierarchical recruitment of phasic dopamine signaling in the striatum during the progression of cocaine use. Proc. Natl. Acad. Sci. USA 109 (2012), 20703–20708.
63. Xia, Y., Driscoll, J.R., Wilbrecht, L., Margolis, E.B., Fields, H.L., Hjelmstad, G.O., Nucleus accumbens medium spiny neurons target non-dopaminergic neurons in the ventral tegmental area. J. Neurosci. 31 (2011), 7811–7816.