Dopamine neurons learn to encode the long-term value of multiple future rewards

Proceedings of the National Academy of Sciences of the United States of America

Volumn 108, Issue 37, 2011, Pages 15462-15467

  Enomoto, Kazuki ^a,b   Matsumoto, Naoyuki ^a,c   Nakai, Sadamu ^a,d   Satoh, Takemasa ^a,e   Sato, Tatsuo K ^a,f   Ueda, Yasumasa ^a   Inokawa, Hitoshi ^a   Haruno, Masahiko ^b,g   Kimura, Minoru ^a,b  

^a KYOTO PREFECTURAL UNIVERSITY OF MEDICINE   (Japan)

^b TAMAGAWA UNIVERSITY   (Japan)

^c PREFECTURAL UNIVERSITY OF KUMAMOTO   (Japan)

^d TOKYO UNIVERSITY OF SCIENCE   (Japan)

^e TOTTORI UNIVERSITY   (Japan)

^f UNIVERSITY COLLEGE LONDON   (United Kingdom)

^g NATIONAL INSTITUTE OF INFORMATION AND COMMUNICATIONS TECHNOLOGY   (Japan)

Author keywords

Basal ganglia; Decision making; Primate; Temporal difference learning

Indexed keywords

ANIMAL EXPERIMENT; ARTICLE; BEHAVIOR; DECISION MAKING; DOPAMINERGIC NERVE CELL; EPISODIC MEMORY; EXPECTATION; EXPERIMENTAL MONKEY; KNOWLEDGE; LICKING; MOTIVATION; MULTIPLE CHOICE TEST; NONHUMAN; PRIORITY JOURNAL; REINFORCEMENT; REWARD;

ANIMALS; ANTICIPATION, PSYCHOLOGICAL; BEHAVIOR, ANIMAL; DOPAMINE; HAPLORHINI; LEARNING; NEURONS; REWARD; TIME FACTORS;

PRIMATES;

EID: 80053075057     PISSN: 00278424     EISSN: 10916490     Source Type: Journal    
DOI: 10.1073/pnas.1014457108     Document Type: Article

References (34)

1. Sutton RS, Barto AG (1998) Reinforcement Learning (MIT Press, Cambridge, MA).
2. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898-1902. (Pubitemid 36356659)
3. Morris G, Arkadir D, Nevet A, Vaadia E, Bergman H (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43:133-143. (Pubitemid 38952825)
4. Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9:1057-1063. (Pubitemid 44264919)
5. Satoh T, Nakai S, Sato T, Kimura M (2003) Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci 23:9913-9923. (Pubitemid 37371606)
6. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129-141. (Pubitemid 40922855)
7. Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O (2004) Dopamine neurons can represent context-dependent prediction error. Neuron 41:269-280. (Pubitemid 38125187)
8. Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593-1599. (Pubitemid 27120526)
9. Redgrave P, Prescott TJ, Gurney K (1999) Is the short-latency dopamine response too short to signal reward error? Trends Neurosci 22:146-151. (Pubitemid 29166781)
10. Matsumoto M, Hikosaka O (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837-841.
11. Roesch MR, Taylor AR, Schoenbaum G (2006) Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51:509-520. (Pubitemid 44189799)
12. Kim S, Hwang J, Lee D (2008) Prefrontal coding of temporally discounted values during intertemporal choice. Neuron 59:161-172. (Pubitemid 351905851)
13. Tsujimoto S, Sawaguchi T (2005) Neuronal activity representing temporal prediction of reward in the primate prefrontal cortex. J Neurophysiol 93:3687-3692. (Pubitemid 40720937)
14. Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837-7846.
15. Roesch MR, Calu DJ, Schoenbaum G (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci 10:1615-1624. (Pubitemid 350175882)
16. Tanaka SC, et al. (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887-893.
17. McClure SM, Laibson DI, Loewenstein G, Cohen JD (2004) Separate neural systems value immediate and delayed monetary rewards. Science 306:503-507. (Pubitemid 39372453)
18. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310:1337-1340. (Pubitemid 41698867)
19. Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58:451-463. (Pubitemid 351615183)
20. Pasquereau B, et al. (2007) Shaping of motor responses by incentive values through the basal ganglia. J Neurosci 27:1176-1183. (Pubitemid 46203326)
21. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936-1947. (Pubitemid 26145969)
22. Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11:966-973.
23. Bromberg-Martin ES, Hikosaka O (2009) Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63:119-126.
24. Ravel S, Richmond BJ (2006) Dopamine neuronal responses in monkeys performing visually cued reward schedules. Eur J Neurosci 24:277-290. (Pubitemid 44050686)
25. Kacelnik A (1997) Normative and descriptive models of decision making: time discounting and risk sensitivity. Ciba Found Symp 208:51-67.
26. Kahneman D, Tversky A (1979) Prospect theory: An analysis of decision under risk. Econometrica 47:263-291.
27. Mazur JE (1987) An adjusting procedure for studying delayed reinforcement. Quantitative Analyses of Behavior, eds Commons ML, Mazur JE, Nevin JA, Rachlin H (Lawrence Erlbaum, Hillsdale, NJ),Vol 5.
28. Kable JW, Glimcher PW (2010) An "as soon as possible" effect in human intertemporal decision making: behavioral evidence and neural mechanisms. J Neurophysiol 103:2513-2531.
29. Louie K, Glimcher PW (2010) Separating value from choice: Delay discounting activity in the lateral intraparietal area. J Neurosci 30:5498-5507.
30. Williams J, Dayan P (2005) Dopamine, learning, and impulsivity: A biological account of attention-deficit/hyperactivity disorder. J Child Adolesc Psychopharmacol 15:160-179. (Pubitemid 40734097)
31. Cardinal RN, Winstanley CA, Robbins TW, Everitt BJ (2004) Limbic corticostriatal systems and delayed reinforcement. Ann N Y Acad Sci 1021:33-50. (Pubitemid 38879608)
32. Denk F, et al. (2005) Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort. Psychopharmacology (Berl) 179:587-596. (Pubitemid 40674759)
33. Pine A, Shiner T, Seymour B, Dolan RJ (2010) Dopamine, time, and impulsivity in humans. J Neurosci 30:8888-8896.
34. Alexander WH, Brown JW (2010) Hyperbolically discounted temporal difference learning. Neural Comput 22:1511-1527.