Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework

eLife

Volumn 5, Issue MARCH2016, 2016, Pages

  Sadacca, Brian F ^a   Jones, Joshua L ^a   Schoenbaum, Geoffrey ^a,b,c  

^a NATIONAL INSTITUTE ON DRUG ABUSE   (United States)

^b UNIVERSITY OF MARYLAND SCHOOL OF MEDICINE   (United States)

^c JOHNS HOPKINS UNIVERSITY SCHOOL OF MEDICINE   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINERGIC NERVE CELL; ERROR; MESENCEPHALON; MODEL; PREDICTION; REWARD; ANIMAL; ANIMAL BEHAVIOR; ASSOCIATION; BIOLOGICAL MODEL; FEMALE; INSTRUMENTAL CONDITIONING; LONG EVANS RAT; MALE; PHYSIOLOGY;

ANIMALS; BEHAVIOR, ANIMAL; CONDITIONING, OPERANT; CUES; DOPAMINERGIC NEURONS; FEMALE; MALE; MESENCEPHALON; MODELS, NEUROLOGICAL; RATS, LONG-EVANS; REWARD;

EID: 84964211344     PISSN: None     EISSN: 2050084X     Source Type: Journal    
DOI: 10.7554/eLife.13665     Document Type: Article

References (57)

1. Bayer HM, Glimcher PW. 2005. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129–141. doi: 10.1016/j.neuron.2005.05.020
2. Brogden WJ. 1939. Sensory pre-conditioning. Journal of Experimental Psychology 25:323–332. doi: 10.1037/h0058944
3. Bromberg-Martin ES, Matsumoto M, Hikosaka O. 2010a. Dopamine in Motivational Control: Rewarding, Aversive, and Alerting. Neuron 68:815–834. doi: 10.1016/j.neuron.2010.11.022
4. Bromberg-Martin ES, Matsumoto M, Hong S, Hikosaka O. 2010b. A pallidus-habenula-dopamine pathway signals inferred stimulus values. Journal of Neurophysiology 104:1068–1076. doi: 10.1152/jn.00158.2010
5. Bush RR, Mosteller F. 1951. A mathematical model for simple learning. Psychological Review 58:313–323. doi: 10.1037/h0054388
6. Chang CY, Esber GR, Marrero-Garcia Y, Yau H-J, Bonci A, Schoenbaum G. 2016. Brief optogenetic inhibition of dopamine neurons mimics endogenous negative reward prediction errors. Nature Neuroscience 19:111–116. doi: 10.1038/nn.4191
7. Clark JJ, Hollon NG, Phillips PEM. 2012. Pavlovian valuation systems in learning and decision making. Current Opinion in Neurobiology 22:1054–1061. doi: 10.1016/j.conb.2012.06.004
8. Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N. 2012. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482:85–88. doi: 10.1038/nature10754
9. Cone JJ, Fortin SM, McHenry JA, Stuber GD, McCutcheon JE, Roitman MF. 2016. Physiological state gates acquisition and expression of mesolimbic reward prediction signals. Proceedings of the National Academy of Sciences of the United States of America 113:1943–1948. doi: 10.1073/pnas.1519643113
10. Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ. 2011. Model-based influences on humans’ choices and striatal prediction errors. Neuron 69:1204–1215. doi: 10.1016/j.neuron.2011.02.027
11. Daw ND, Niv Y, Dayan P. 2005. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8:1704–1711. doi: 10.1038/nn1560
12. Deserno L, Huys QJM, Boehme R, Buchert R, Heinze H-J, Grace AA, Dolan RJ, Heinz A, Schlagenhauf F. 2015. Ventral striatal dopamine reflects behavioral and neural signatures of model-based control during sequential decision making. Proceedings of the National Academy of Sciences of the United States of America 112:1595–1600. doi: 10.1073/pnas.1417219112
13. Doll BB, Bath KG, Daw ND, Frank MJ. 2016. Variability in dopamine genes dissociates model-based and model-free reinforcement learning. Journal of Neuroscience 36:1211–1222. doi: 10.1523/JNEUROSCI.1901-15.2016
14. Doll BB, Simon DA, Daw ND. 2012. The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology 22:1075–1081. doi: 10.1016/j.conb.2012.08.003
15. Eshel N, Bukwich M, Rao V, Hemmelder V, Tian J, Uchida N. 2015. Arithmetic and local circuitry underlying dopamine prediction errors. Nature 525:243–246. doi: 10.1038/nature14855
16. Glimcher PW. 2011. Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences of the United States of America 108: 15647–15654. doi: 10.1073/pnas.1014269108
17. Hall G. 1996. Learning about associatively activated stimulus representations: Implications for acquired equivalence and perceptual learning. Animal Learning & Behavior 24:233–255. doi: 10.3758/BF03198973
18. Hamid AA, Pettibone JR, Mabrouk OS, Hetrick VL, Schmidt R, Vander Weele CM, Kennedy RT, Aragona BJ, Berke JD. 2016. Mesolimbic dopamine signals the value of work. Nature Neuroscience 19:117–126. doi: 10.1038/nn.4173
19. Hart AS, Rutledge RB, Glimcher PW, Phillips PEM. 2014. Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. Journal of Neuroscience 34:698–704. doi: 10.1523/JNEUROSCI.2489-13.2014
20. Holland PC, Straub JJ. 1979. Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning. Journal of Experimental Psychology 5:65–78. doi: 10.1037/0097-7403.5.1.65
21. Holland PC. 1990. Event representation in Pavlovian conditioning: Image and action. Cognition 37:105–131. doi: 10.1016/0010-0277(90)90020-K
22. Hollerman JR, Schultz W. 1998. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1:304–309. doi: 10.1038/1124
23. Howe MW, Tierney PL, Sandberg SG, Phillips PEM, Graybiel AM. 2013. Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500:575–579. doi: 10.1038/nature12475
24. Jin X, Costa RM. 2010. Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 466: 457–462. doi: 10.1038/nature09263
25. Jo YS, Lee J, Mizumori SJY. 2013. Effects of prefrontal cortical inactivation on neural activity in the ventral tegmental area. Journal of Neuroscience 33:8159–8171. doi: 10.1523/JNEUROSCI.0118-13.2013
26. Jones JL, Esber GR, McDannald MA, Gruber AJ, Hernandez A, Mirenzi A, Schoenbaum G. 2012. Orbitofrontal cortex supports behavior and learning using inferred but not cached values. Science 338:953–956. doi: 10.1126/science.1227489
27. Kakade S, Dayan P. 2002. Dopamine: generalization and bonuses. Neural Networks 15:549–559. doi: 10.1016/S0893-6080(02)00048-5
28. Kurth-Nelson Z, Barnes G, Sejdinovic D, Dolan R, Dayan P. 2015. Temporal structure in associative retrieval. eLife 4. doi: 10.7554/eLife.04919
29. Lak A, Stauffer WR, Schultz W. 2014. Dopamine prediction error responses integrate subjective value from different reward dimensions. Proceedings of the National Academy of Sciences of the United States of America 111:2343–2348. doi: 10.1073/pnas.1321596111
30. Mackintosh NJ. 1974. The Psychology of Animal Learning. London: Academic Press.
31. Margolis EB, Lock H, Hjelmstad GO, Fields HL. 2006. The ventral tegmental area revisited: is there an electrophysiological marker for dopaminergic neurons? The Journal of Physiology 577:907–924. doi: 10.1113/jphysiol.2006.117069
32. Martinez V, Walton ME, Gan JO, Phillips PEM. 2008. Phasic dopamine transmission following state-based reinforcer devaluation in a dual-reward detection task. Society for Neuroscience Abstracts.
33. Matsumoto M, Hikosaka O. 2009. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837–841. doi: 10.1038/nature08028
34. Mirenowicz J, Schultz W. 1994. Importance of unpredictability for reward responses in primate dopamine neurons. Journal of Neurophysiology 72:1024–1027.
35. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H. 2006. Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience 9:1057–1063. doi: 10.1038/nn1743
36. Niv Y, Daw ND, Joel D, Dayan P. 2007. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology 191:507–520. doi: 10.1007/s00213-006-0502-4
37. Niv Y, Schoenbaum G. 2008. Dialogues on prediction errors. Trends in Cognitive Sciences 12:265–272. doi: 10.1016/j.tics.2008.03.006
38. Pan WX, Schmidt R, Wickens JR, Hyland BI. 2005. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. Journal of Neuroscience 25:6235–6242. doi: 10.1523/JNEUROSCI.1478-05.2005
39. Rescorla RA, Freberg L. 1978. The extinction of within-compound flavor associations. Learning and Motivation 9: 411–427. doi: 10.1016/0023-9690(78)90003-6
40. Rescorla RA, Wagner AR. 1972. Classical Conditioning II: Current Research. In: Theory A. H, Black W. F. Prokasy eds New York: Appleton-Century-Crofts 64–99.
41. Roesch MR, Calu DJ, Schoenbaum G. 2007. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience 10:1615–1624. doi: 10.1038/nn2013
42. Schultz W, Carelli RM, Wightman RM. 2015. Phasic dopamine signals: from subjective reward value to formal economic utility. Current Opinion in Behavioral Sciences 5:147–154. doi: 10.1016/j.cobeha.2015.09.006
43. Schultz W, Dayan P, Montague PR. 1997. A neural substrate of prediction and reward. Science 275:1593–1599. doi: 10.1126/science.275.5306.1593
44. Schultz W. 2002. Getting formal with dopamine and reward. Neuron 36:241–263. doi: 10.1016/S0896-6273(02)00967-4
45. Sharp ME, Foerde K, Daw ND, Shohamy D. 2016. Dopamine selectively remediates’model-based’ reward learning: a computational approach. Brain 139:355–364. doi: 10.1093/brain/awv347
46. Stalnaker TA, Cooch NK, Schoenbaum G. 2015. What the orbitofrontal cortex does not do. Nature Neuroscience 18:620–627. doi: 10.1038/nn.3982
47. Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH. 2013. A causal link between prediction errors, dopamine neurons and learning. Nature Neuroscience 16:966–973. doi: 10.1038/nn.3413
48. Sutton RS. 1988. Learning to predict by the methods of temporal differences. Machine Learning 3:9–44. doi: 10.1007/BF00115009
49. Takahashi YK, Chang CY, Lucantonio F, Haney RZ, Berg BA, Yau H-J, Bonci A, Schoenbaum G. 2013. Neural estimates of imagined outcomes in the orbitofrontal cortex drive behavior and learning. Neuron 80:507–518. doi: 10.1016/j.neuron.2013.08.008
50. Takahashi YK, Roesch MR, Stalnaker TA, Haney RZ, Calu DJ, Taylor AR, Burke KA, Schoenbaum G. 2009. The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron 62:269–280. doi: 10.1016/j.neuron.2009.03.005
51. Takahashi YK, Roesch MR, Wilson RC, Toreson K, O’Donnell P, Niv Y, Schoenbaum G. 2011. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nature Neuroscience 14:1590–1597. doi: 10.1038/nn.2957
52. Tobler PN, Dickinson A, Schultz W. 2003. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. Journal of Neuroscience 23:10402–10410.
53. Waelti P, Dickinson A, Schultz W. 2001. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43–48. doi: 10.1038/35083500
54. Ward-Robinson J, Hall G. 1996. Backward sensory preconditioning. Journal of Experimental Psychology: Animal Behavior Processes 22:395–404. doi: 10.1037/0097-7403.22.4.395
55. Wimmer GE, Shohamy D. 2012. Preference by Association: How Memory Mechanisms in the Hippocampus Bias Decisions. Science 338:270–273. doi: 10.1126/science.1223252
56. Wunderlich K, Smittenaar P, Dolan RJ. 2012. Dopamine enhances model-based over model-free choice behavior. Neuron 75:418–424. doi: 10.1016/j.neuron.2012.03.042
57. Young AMJ, Ahier RG, Upton RL, Joseph MH, Gray JA. 1998. Increased extracellular dopamine in the nucleus accumbens of the rat during associative learning of neutral stimuli. Neuroscience 83:1175–1183. doi: 10.1016/S0306-4522(97)00483-1