Temporal-difference reinforcement learning with distributed representations

PLoS ONE

Volumn 4, Issue 10, 2009, Pages

  Kurth Nelson, Zeb ^a   Redish, A David ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

ARTICLE; BEHAVIORAL RESEARCH; CONDITIONING; CONTROLLED STUDY; DELAY CONDITIONING; LEARNING ALGORITHM; MATHEMATICAL COMPUTING; REINFORCEMENT; REWARD; STIMULUS RESPONSE; TEMPORAL DIFFERENCE ALGORITHM; TRACE CONDITIONING; ALGORITHM; ANIMAL; ARTIFICIAL NEURAL NETWORK; BIOLOGICAL MODEL; COMPUTER PROGRAM; COMPUTER SIMULATION; HUMAN; LEARNING; PROBABILITY; REPRODUCIBILITY; STATISTICAL MODEL; TIME;

ANIMALIA;

ALGORITHMS; ANIMALS; COMPUTER SIMULATION; CONDITIONING (PSYCHOLOGY); HUMANS; LEARNING; MARKOV CHAINS; MODELS, NEUROLOGICAL; MODELS, STATISTICAL; NEURAL NETWORKS (COMPUTER); REINFORCEMENT (PSYCHOLOGY); REPRODUCIBILITY OF RESULTS; REWARD; SOFTWARE; TIME FACTORS;

EID: 70449382577     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0007362     Document Type: Article

References (113)

1. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience 16: 1936-1947.
2. Schultz W, Dayan P, Montague R (1997) A neural substrate of prediction and reward. Science 275: 1593-1599.
3. Doya K (2000) Metalearning, neuromodulation, and emotion. In: Hatano G, Okada N, Tanabe H, eds. Affective Minds, Elsevier.
4. Sutton RS, Barto AG (1998) Reinforcement Learning: An introduction. Cambridge MA: MIT Press.
5. Daw ND, Courville AC, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Computation 18: 1637-1677.
6. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychological Review 114: 784-805.
7. Sutton RS, ed (1992) Special issue on reinforcement learning, volume 8(3/ 4) of Machine Learning. Boston: Kluwer Academic Publishers.
8. Daw ND (2003) Reinforcement learning models of the dopamine system and their behavioral implications. Ph.D. thesis, Carnegie Mellon University.
9. Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokesy WF, eds. Classical Conditioning II: Current Research and Theory. New York: Appleton Century Crofts. pp 64-99.
10. Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review 88: 135-170.
11. Barto AG (1995) Adaptive critics and the basal ganglia. In: Houk JC, Davis JL, Beiser DG, eds. Models of Information Processing in the Basal Ganglia. Cambridge MA: MIT Press. pp 215-232.
12. Schultz W, Dickinson A (2000) Neuronal coding of prediction errors. Annual Review of Neuroscience 23: 473-500.
13. Schultz W (2002) Getting formal with dopamine and reward. Neuron 36: 241-263.
14. Redish AD (2004) Addiction as a computational process gone awry. Science 306: 1944-1947.
15. Kakade S, Dayan P (2002) Dopamine: generalization and bonuses. Neural Networks 15: 549-599.
16. O'Doherty JP, PeterDayan, KF, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38: 329-337.
17. O'Doherty JP (2004) Reward representations and reward-related learning in the human brain: insights from neuroimaging. Current Opinion in Neurobiology 14: 769-776.
18. Bayer HM, Glimcher P (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47: 129-141.
19. Pan WX, Schmidt R, Wickens JR, Hyland BI (2005) Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network. J Neurosci 25: 6235-6242.
20. Stuber GD, Wightman RM, Carelli RM (2005) Extinction of cocaine self-administration reveals functionally and temporally distinct dopaminergic signals in the nucleus accumbens. Neuron 46: 661-669.
21. Day JJ, Roitman MF, Wightman RM, Carelli RM (2007) Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nature Neuroscience 10: 1020-1028.
22. Bayer HM, Lau B, Glimcher PW (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98: 1428-1439.
23. Doya K, Samejima K, Katagiri KI, Kawato M (2002) Multiple model-based reinforcement learning. Neural Computation 14: 1347-1369.
24. Bertin M, Schweighofer N, Doya K (2007) Multiple model-based reinforcement learning explains dopamine neuronal activity. Neural Networks 20: 668-675.
25. Ludvig EA, Sutton RS, Kehoe EJ (2008) Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation 20: 3034-3054.
26. Ludvig EA, Sutton RS, Verbeek E, Kehoe EJ (2009) A computational model of hippocampal function in trace conditioning. In: Koller D, Schuurmans D, Bengio Y, Bottou L, eds. Advances in Neural Information Processing Systems 21. pp 993-1000.
27. Alexander GE, DeLong MR, Strick PL (1986) Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual Reviews Neuroscience 9: 357-381.
28. Strick PL, Dum RP, Picard N (1995) Macro-organization of the circuts connecting the basal ganglia with the cortical motor areas. In: Houk JC, Davis JL, Beiser DG, eds. Models of Information Processing in the Basal Ganglia, MIT Pres. pp 117-130.
29. Haber SN, Fudge JL, McFarland NR (2000) Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. Journal of Neuroscience 20: 2369-2382.
30. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, et al. (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nature Neuroscience 7: 887-893.
31. Schweighofer N, Bertin M, Shishida K, Okamoto Y, Tanaka SC, et al. (2008) Low-serotonin levels increase delayed reward discounting in humans. Journal of Neuroscience 28: 4528-4532.
32. Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology 67: 145-163.
33. Hollerman JR, Schultz W (1998) Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1: 304-309.
34. Schultz W (1998) Predictive reward signal of dopamine neurons. Journal of Neurophysiology 80: 1-27.
35. Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Current Opinion in Neurobiology 14: 139-147.
36. Stephens DW, Krebs JR (1987) Foraging Theory. Princeton.
37. Redish AD, Kurth-Nelson Z (2010) Neural models of temporal discounting. In: Madden G, Bickel W, eds. Impulsivity: The Behavioral and Neurological Science of Discounting, APA books. pp 123-158.
38. Ainslie G (1992) Picoeconomics. Cambridge Univ Press.
39. Mazur J (1997) Choice, delay, probability and conditioned reinforcement. Animal Learning and Behavior 25: 131-147.
40. Mazur JE (2001) Hyperbolic value addition and general models of animal choice. Psychological Review 108: 96-112.
41. Ainslie G (2001) Breakdown of Will. Cambridge Univ Press.
42. Madden G, Bickel W, Critchfield T, eds (in press) Impulsivity: Theory, Science, and Neuroscience of Discounting. APA books.
43. Bellman R (1958) On a routing problem. Quarterly Journal of Applied Mathematics 16: 87-90.
44. Si J, Barto AG, Powell WB, Wuncsch II D, eds (2004) Handbook of learning and approximate dynamic programming. Wiley: IEEE Press.
45. Rubin DC, Wenzel AE (1996) One hundred years of forgetting: A quantitative description of retention. Psyhcological Review 103: 734-760.
46. Rubin DC, Hinton S, Wenzel A (1999) The precise time course of retention. Journal of Experimental Psychology: Learning, Memory, and Cognition 25: 1161-1176.
47. Herrnstein RJ (1997) The Matching Law. Harvard Univ Press.
48. Madden GJ, Bickel WK, Jacobs EA (1999) Discounting of delayed rewards in opioid-dependent outpatients exponential or hyperbolic discounting functions? Experimental and Clinical Psychopharmacology 7: 284-293.
49. Read D (2001) Is time-discounting hyperbolic or subadditive? Journal of Risk and Uncertainty 23: 5-32.
50. Petry NM, Bickel WK (1998) Polydrug abuse in heroin addicts: a behavioral economic analysis. Addiction 93: 321-335.
51. Mitchell SH (1999) Measures of impulsivity in cigarette smokers and non-smokers. Psychopharmacology 146: 455-464.
52. Odum AL, Madden GJ, Bickel WK (2002) Discounting of delayed health gains and losses by current, never- and ex-smokers of cigarettes. Nicotine and Tobacco Research 4: 295-303.
53. Alessi SM, Petry NM (2003) Pathological gambling severity is associated with impulsivity in a delay discounting procedure. Behavioural Processes 64: 345-354.
54. Reynolds B (2006) A review of delay-discounting research with humans: relations to drug use and gambling. Behavioural Pharmacology 17: 651-667.
55. Shors TJ (2004) Memory traces of trace memories: neurogenesis, synaptogenesis and awareness. Trends in Neurosciences 27: 250-256.
56. Phillips PEM, Stuber GD, Heien MLAV, Wightman RM, Carelli RM (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422: 614-618.
57. Roitman MF, Stuber GD, Phillips PEM, Wightman RM, Carelli RM (2004) Dopamine operates as a subsecond modulator of food seeking. Journal of Neuroscience 24: 1265-1271.
58. Pavlov I (1927) Conditioned Reflexes. Oxford Univ Press.
59. Solomon PR, Schaaf ERV, Thompson RF, Weisz DJ (1986) Hippocampus and trace conditioning of the rabbit's classically conditioned nictitating membrane response. Behavioral Neuroscience 100: 729-744.
60. Beylin AV, Gandhi CC, Wood GE, Talk AC, Matzel LD, et al. (2001) The role of the hippocampus in trace conditioning: Temporal discontinuity or task difficulty? Neurobiology of Learning and Memory 76: 447-461.
61. Thibaudeau G, Potvin O, Allen K, Dore FY, Goulet S (2007) Dorsal, ventral, and complete excitotoxic lesions of the hippocampus in rats failed to impair appetitive trace conditioning. Behavioural Brain Research 185: 9-20.
62. Levy WB, Sanyal A, Rodriguez P, Sullivan DW, Wu XB (2005) The formation of neural codes in the hippocampus: trace conditioning as a prototypical paradigm for studying the random recoding hypothesis. Biol Cybern 92: 409-426.
63. Pastalkova E, Itskov V, Amarasingham A, Buzsaki G (2008) Internally generated cell assembly sequences in the rat hippocampus. Science 321: 1322-1327.
64. Kaplan PS (1984) Bridging temporal gaps between cs and us in autoshaping: A test of a local context hypothesis. Animal Learning and Behavior 12: 142-148.
65. Self D (2003) Dopamine as chicken and egg. Nature 422: 573-574.
66. Mirenowicz J, Schultz W (1996) Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379: 449-451.
67. Suri RE, Schultz W (2001) Temporal difference model reproduces anticipatory neural activity. Neural Computation 13: 841-862.
68. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
69. Mirenowicz J, Schultz W (1994) Importance of unpredictability for reward responses in primate dopamine neurons. Journal of Neurophysiology 72: 1024-1027.
70. Wörgötter F, Porr B (2005) Temporal sequence learning, prediction, and control - a review of different models and their relation to biological mechanisms. Neural Computation 17: 245-319.
71. Daw ND, Kakade S, Dayan P (2002) Opponent interactions between serotonin and dopamine. Neural Networks 15: 603-616.
72. Ainslie G, Monterosso J (2004) Behavior: A marketplace in the brain? Science 306: 421-423.
73. Sozou PD (1998) On hyperbolic discounting and uncertain hazard rates. The Royal Society London B 265: 2015-2020.
74. Kacelnik A (1997) Normative and descriptive models of decision making: time discounting and risk sensitivity. In: Bock GR, Cardew G, eds. Characterizing Human Psychological Adaptations. Chichester UK: Wiley, volume 208 of Ciba Foundation Symposia. pp 51-66. Discussion 67-70.
75. Laibson DI (1996) An economic perspective on addiction and matching. Behavioral and Brain Sciences 19: 583-584.
76. McClure SM, Laibson DI, Loewenstein G, Cohen JD (2004) Separate neural systems value immediate and delayed monetary rewards. Science 306: 503-507.
77. Sanfey AG, Loewenstein G, McClure SM, Cohen JD (2006) Neuroeconomics: cross-currents in research on decision-making. Trends in Cognitive Sciences 10: 108-116.
78. Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ (2001) Impulsive choice induced in rats by lesion of the nucleus accumbens core. Science 292: 2499-2501.
79. Staddon JER, Cerutti DT (2003) Operant conditioning. Annual Reviews of Psychology 54: 115-144.
80. Kalenscher T, Pennartz CMA (2008) Is a bird in the hand worth two in the future? the neuroeconomics of intertemporal decision-making. Progress in Neurobiology 84: 284-315.
81. Gibbon J, Church RM, Fairhurst S, Kacelnik A (1988) Scalar expectancy theory and choice between delayed rewards. Psychological Review 95: 102-114.
82. Gallistel CR, Gibbon J (2000) Time, rate, and conditioning. Psychological Review 107: 289-344.
83. Samejima K, Doya K, Kawato M (2003) Inter-module credit assignment in modular reinforcement learning. Neural Networks 16: 985-994.
84. Wightman RM, Heien MLAV, Wassum KM, Sombers LA, Aragona BJ, et al. (2007) Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. European Journal of Neuroscience 26: 2046-2054.
85. Levy WB (1996) A sequence predicting CA3 is a flexible associator that learns and uses context to solve hippocampal-like tasks. Hippocampus 6: 579-591.
86. Zemel RS, Dayan P, Pouget A (1998) Probabilistic interpretation of population codes. Neural Computation 10: 403-430.
87. Dayan P, Abbott LF (2001) Theoretical Neuroscience. MIT Press.
88. Jackson JC, Redish AD (2003) Detecting dynamical changes within a simulated neural ensemble using a measure of representational quality. Network: Computation in Neural Systems 14: 629-645.
89. Johnson A, Seeland KD, Redish AD (2005) Reconstruction of the postsubiculum head direction signal from neural ensembles. Hippocampus 15: 86-96.
90. Johnson A, Jackson J, Redish AD (2008) Measuring distributed properties of neural representations beyond the decoding of local variables - implications for cognition. In: Hölscher C, Munk MHJ, eds. Mechanisms of information processing in the Brain: Encoding of information in neural populations and networks, Cambridge University Press. pp 95-119.
91. Dayan P (2002) Motivated reinforcement learning. In: Dietterich TG, Becker S, Ghahramani Z, eds. Advances in Neural Information Processing Systems 14. Cambridge, MA: MIT Press.
92. Suri RE, Schultz W (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91: 871-890.
93. Norris JR (1997) Markov Chains. New York: Cambridge University Press.
94. Brémaud P (1999) Markov Chains: Gibbs Fields, Monte Carlo Simulation, and Queues. New York: Springer.
95. Daw ND, Courville AC, Touretzky DS (2002) Dopamine and inference about timing. Proceedings of the Second International Conference on Development and Learning.
96. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty, and TD learning. Behavioral and Brain Functions 1: 6.
97. Badtke SJ, Duff MO (1995) Reinforcement-learning methods for continuoustime Markov decision problems. In: Tesauro G, Touretzky D, Leen T, eds. Advances in Neural Information Processing 7, MIT Press.
98. Das T, Gosavi A, Mahadevan S, Marchalleck N (1999) Solving semi-markov decision problems using average reward reinforcement learning. Management Science 45: 575-596.
99. Schultz W, Romo R, Ljungberg T, Mirenowicz J, hollerman JR, et al. (1995) Reward-related signals carried by dopamine neurons. In: Houk JC, Davis JL, Beiser DG, eds. Models of Information Processing in the Basal Ganglia. Cambridge MA: MIT Press. pp 233-248.
100. Fiorillo CD, Tobler PN, Schultz W (2005) Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropogating TD errors. Behavioral and Brain Functions 1: 7.
101. Cheer JF, Aragona BJ, Heien MLAV, Seipel AT, Carelli RM, et al. (2007) Coordinated accumbal dopamine release and neural activity drive goal-directed behavior. Neuron 54: 237-244.
102. Mackintosh NJ (1974) The Psychology of Animal Learning. Academic Press.
103. Fiorillo CD, Newsome WT, Schultz W (2008) The temporal precision of reward prediction in dopamine neurons. Nature Neuroscience 11: 966-973.
104. Doya K (1999) What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex? Neural networks 12: 961-974.
105. Doya K (2000) Complementary roles of basal ganglia and cerebellum in learning and motor control. Current Opinion in Neurobiology 10: 732-739.
106. Samejima K, Ueda Y, Doya K, Kimura M (2005) Representation of action-specific reward values in the striatum. Science 310: 1337-1340.
107. Kawato M, Samejima K (2007) Efficient reinforcement learning: computational theories, neuroscience and robotics. Current Opinion in Neurobiology 17: 205-212.
108. Alexander GE, Crutcher MD (1990) Functional architecture of basal ganglia circuits: Neural substrates of parallel processing. Trends in Neurosciences 13: 266-271.
109. Graybiel AM, Flaherty AW, Giménez-Amaya JM (1991) Striosomes and matrisomes. In: Bernardi G, Carpenter MB, Di Chiara G, eds. The Basal Ganglia III, Plenum.
110. Vuchinich RE, Simpson CA (1998) Hyperbolic temporal discounting in social drinkers and problem drinkers. Experimental and Clinical Psychopharmacology 6: 292-305.
111. Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards. theory, experiments, and possible neural mechanisms. Annals of the New York Academy of Sciences 1104: 289-300.
112. Tanaka SC, Schweighofer N, Asahi S, Okamoto Y, Doya K (2004) An fMRI study of the delay discounting of reward after tryptophan depletion and loading. 2: reward-expectation. Society for Neuroscience Abstracts.
113. Schweighofer N, Shishida K, Han CE, Yamawaki YOSCTS, Doya K (2006) Humans can adopt optimal discounting strategy under real-time constraints. PLoS Computational Biology 2: e152.