Short-term memory traces for action bias in human reinforcement learning

Brain Research

Volumn 1153, Issue 1, 2007, Pages 111-121

  Bogacz, Rafal ^a   McClure, Samuel M ^a,b   Li, Jian ^b   Cohen, Jonathan D ^a   Montague, P Read ^b  

^a PRINCETON UNIVERSITY   (United States)

^b BAYLOR COLLEGE OF MEDICINE   (United States)

Author keywords

Dopamine; Eligibility traces; Reinforcement learning

Indexed keywords

ARTICLE; BEHAVIOR; BRAIN FUNCTION; CONTROLLED STUDY; DECISION MAKING; DOPAMINE RELEASE; HUMAN; HUMAN EXPERIMENT; LEARNING; MATHEMATICAL COMPUTING; MESENCEPHALON; MODEL; NERVE CELL PLASTICITY; NEUROSCIENCE; PRIORITY JOURNAL; REINFORCEMENT; SHORT TERM MEMORY; SIMULATION; TASK PERFORMANCE;

BIAS (EPIDEMIOLOGY); HUMANS; MEMORY, SHORT-TERM; MODELS, NEUROLOGICAL; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY);

ANIMALIA;

EID: 34248999741     PISSN: 00068993     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.brainres.2007.03.057     Document Type: Article

References (52)

1. Abbott L.F., and Nelson S.B. Synaptic plasticity: taming the beast. Nat. Neurosci. Suppl. 3 (2000) 1178-1183
2. Barto A.G., Sutton R.S., and Brouwer P.S. Associative search network: a reinforcement learning associative memory. Biol. Cybern. 40 (1981) 201-211
3. Bayer H.M., and Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47 (2005) 129-141
4. Benzion U., Rapoport A., and Yagil J. Discount rates inferred from decisions: an experimental study. Manag. Sci. 35 (1989) 270-284
5. Berns G.S., McClure S.M., Pagnoni G., and Montague P.R. Predictability modulates human brain response to reward. J. Neurosci. 21 (2001) 2793-2798
6. Berridge K.C., and Robinson T.E. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience?. Brain Res. Rev. 28 (1998) 309-369
7. Bogacz R., Brown E., Moehlis J., Holmes P., and Cohen J.D. The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced choice tasks. Psychol. Rev. 113 (2006) 700-765
8. Breiter H.C., and Rosen B.R. Functional magnetic resonance imaging of brain reward circuitry in the human. Ann. N. Y. Acad. Sci. 877 (1999) 523-547
9. Daw N.D., Niv Y., and Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8 (2005) 1704-1711
10. Daw N.D., O'Doherty J.P., Dayan P., Seymour B., and Dolan R.J. Cortical substrates for exploratory decisions in humans. Nature 44 (2006) 876-879
11. Dayan P., Kakade S., and Montague P.R. Learning and selective attention. Nat. Neurosci. 3 (2000) 1218-1223
12. Egelman D.M., Person C., and Montague P.R. A computational role for dopamine delivery in human decision-making. J. Cogn. Neurosci. 10 (1998) 623-630
13. Gold J.I., and Shadlen M.N. Neural computations that underlie decisions about sensory stimuli. Trends Cogn. Sci. 5 (2001) 10-16
14. Gold J.I., and Shadlen M.N. Banburismus and the brain: decoding the relationship between sensory stimuli, decisions and reward. Neuron 36 (2002) 299-308
15. Herrnstein R.J. Melioration as behavioral dynamism. In: Commons M.L., Herrnstein R.J., and Rachlin H. (Eds). Quantitative Analyses of Behavior. Matching and Maximizing Accounts vol. II (1982), Ballinger Publishing Co, Cambridge, MA p.^pp.
16. Herrnstein R.J. Rational choice theory: necessary but not sufficient. Am. Psychol. 45 (1990) 356-367
17. Izhikevich, E.M., in press. Solving the distal reward problem through linkage of STDP and dopamine signaling, Cereb. Cortex (doi:10.1093/cercor/bhl152).
18. Laming D.R.J. Information Theory of Choice Reaction Time (1968), Wiley, New York
19. Li J., McClure S.M., King-Casas B., and Montague P.R. Policy adjustment in a dynamic economic game. PLoS ONE 1 (2006) e103
20. Manes F., Sahakian B., Clark L., Rogers R., Antoun N., Aitken M., and Robbins T. Decision-making processes following damage to the prefrontal cortex. Brain 125 (2002) 624-639
21. McClure S.M., Daw N.D., and Montague P.R. A computational substrate for incentive salience. Trends Neurosci. 26 (2003) 423-428
22. Michael A.C., Ikeda M., and Justice Jr. J.B. Mechanisms contributing to the recovery of striatal releasable dopamine following MFB stimulation. Brain Res. 421 (1987) 325-335
23. Montague P.R., and Berns G.S. Neural economics and the biological substrates of valuation. Neuron 36 (2002) 265-284
24. Montague P.R., and Sejnowski T.J. The predictive brain: temporal coincidence and temporal order in synaptic learning mechanisms. Learn. Mem. (1994) 1-13
25. Montague P.R., Dayan P., and Sejnowski T.J. Foraging in an uncertain environment using predictive Hebbian learning. Adv. Neural Inf. Process. Syst. 6 (1994) 598-605
26. Montague P.R., Dayan P., and Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16 (1996) 1936-1947
27. Montague P.R., McClure S.M., Baldwin P.R., Phillips P.E.M., Budygin E.A., Stuber G.D., Kilpatrick M.R., and Wightman R.M. Plasticity in neuromodulator release: dynamic control of dopamine delivery in freely moving animals. J. Neurosci. 24 (2004) 1754-1759
28. Montague P.R., King-Casas B., and Cohen J.D. Imaging valuation models in human choice. Annu. Rev. Neurosci. 29 (2006) 417-448
29. Nedler J.A., and Mead R. A simple method for function minimization. Comput. J. 7 (1965) 308-313
30. Olds J. Hypothalamic substrates of reward. Psychol. Rev. 42 (1962) 554-604
31. Pan W.X., Schmidt R., Wickens J.R., and Hyland B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25 (2005) 6235-6242
32. Ratcliff R. A theory of memory retrieval. Psychol. Rev. 83 (1978) 59-108
33. Ratcliff R. Modeling response signal and response time data. Cogn. Psychol. 53 (2006) 195-237
34. Ratcliff R., and Smith P.L. A comparison of sequential sampling models for two-choice reaction time. Psychol. Rev. 111 (2004) 333-367
35. Ratcliff R., Van Zandt T., and McKoon G. Connectionist and diffusion models of reaction time. Psychol. Rev. 106 (1999) 261-300
36. Ratcliff R., Cherian A., and Segraves M. A comparison of macaques behavior and superior colliculus neuronal activity to predictions from models of two-choice decisions. J. Neurophysiol. 90 (2003) 1392-1407
37. Raymond J.L., and Lisberger S.G. Neural learning rules for vestibulo-ocular reflex. J. Neurosci. 18 (1998) 9112-9129
38. Reynolds J.N., and Wickens J. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 15 (2002) 507-521
39. Reynolds J.N., Hyland B.I., and Wickens J.R. A cellular mechanism of reward-related learning. Nature 413 (2001) 67-70
40. Rolls E.T. The orbitofrontal cortex and reward. Cereb. Cortex 10 (2000) 284-294
41. Schall J.D. Neural basis of deciding, choosing and acting. Nat. Rev., Neurosci. 2 (2001) 33-42
42. Schultz W., Dayan P., and Montague P.R. A neural substrate of prediction and reward. Science 275 (1997) 1593-1599
43. Shadlen M.N., and Newsome W.T. Motion perception: seeing and deciding. Proc. Natl. Acad. Sci. U. S. A. 93 (1996) 628-633
44. Shadlen M.N., and Newsome W.T. Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey. J. Neurophysiol. 86 (2001) 1916-1936
45. Shizgal P. On the neural computation of utility: implications from studies of brain stimulation reward. In: Kahneman D., Diener E., and Schwarz N. (Eds). Foundations of Hedonic Psychology: Scientific Perspectives on Enjoyment and Suffering, Russell Sage Foundation (1999)
46. Singh S.P., and Sutton R.S. Reinforcement learning with replacing eligibility traces. Mach. Learn. 22 (1996) 123-158
47. Stone M. Models for choice reaction time. Psychometrika 25 (1960) 251-260
48. Sutton R.S., and Barto A.G. Reinforcement Learning (1998), MIT Press, Cambridge, MA
49. Wald A., and Wolfowitz J. Optimum character of the sequential probability ratio test. Ann. Math. Stat. 19 (1948) 326-333
50. West A.R., and Grace A.A. Striatal nitric oxide signaling regulates the neuronal activity of midbrain dopamine neurons in vivo. J. Neurophysiol. 83 (2000) 1796-1808
51. Wickens J., and Kotter R. Cellular models of reinforcement. In: Houk J.C., Davis J.L., and Beiser D.G. (Eds). Models of Information Processing in Basal Ganglia (1995), MIT Press, Cambridge, MA pp. 187--214
52. Wickens J.R., Reynolds J.N., and Hyland B.I. Neural mechanisms of reward-related motor learning. Curr. Opin. Neurobiol. 13 (2003) 685-690