Reinforcement learning with Marr

Current Opinion in Behavioral Sciences

Volumn 11, Issue , 2016, Pages 67-73

  Niv, Yael ^a   Langdon, Angela ^a  

^a PRINCETON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

BASAL GANGLION; CONCEPTUAL FRAMEWORK; CORPUS STRIATUM; DECISION MAKING; HIDDEN MARKOV MODEL; LEARNING; LEARNING ALGORITHM; NEUROMODULATION; PSYCHOLOGY; REINFORCEMENT; REVIEW;

EID: 84973375982     PISSN: None     EISSN: 23521546     Source Type: Journal    
DOI: 10.1016/j.cobeha.2016.04.005     Document Type: Review

References (103)

1. Marr D., Poggio T. From understanding computation to understanding neural circuitry. Neurosci Res Program Bull 1977, 15:470-488.
2. Sutton R.S., Barto A.G. Reinforcement learning: an introduction 1998, MIT Press.
3. Sutton R.S. Learning to predict by the methods of temporal differences. Mach Learn 1988, 3:9-44.
4. Niv Y., Schoenbaum G. Dialogues on prediction errors. Trends Cogn Sci 2008, 12:265-272.
5. Niv Y. Reinforcement learning in the brain. J Math Psychol 2009, 53:139-154.
6. Houk J.C., Adams J.L., Barto A.G. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of information processing in the basal ganglia 1995, 249-270. MIT Press. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
7. O'Reilly R.C., Frank M.J. Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Computat 2006, 18:283-328.
8. Redgrave P., Prescott T.J., Gurney K. The basal ganglia: a vertebrate solution to the selection problem?. Neuroscience 1999, 89:1009-1023.
9. Montague P.R., Dayan P., Sejnowski T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 1996, 16:1936-1947.
10. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
11. Barto A.G. Adaptive critics and the basal ganglia. Models of information processing in the basal ganglia 1995, 215-232. MIT Press. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
12. Kang M.J., Hsu M., Krajbich I.M., Loewenstein G., McClure S.M., Wang J.T., Camerer C.F. The wick in the candle of learning epistemic curiosity activates reward circuitry and enhances memory. Psychol Sci 2009, 20:963-973.
13. Loewenstein G. The psychology of curiosity: a review and reinterpretation. Psychol Bull 1994, 116:75.
14. Schmidhuber J. A possibility for implementing curiosity and boredom in model-building neural controllers. From animals to animats: proceedings of the first international conference on simulation of adaptive behavior 1991.
15. Oudeyer P.-Y., Kaplan F. What is intrinsic motivation? A typology of computational approaches. Front Neurorobot 2007, 1:6.
16. Barto A.G. Intrinsic motivation and reinforcement learning. Intrinsically motivated learning in natural and artificial systems 2013, 17-47. Springer. G. Baldassarre, M. Mirolli (Eds.).
17. Şimşek Ö., Barto A.G. An intrinsic reward mechanism for efficient exploration. Proceedings of the 23rd international conference on Machine learning 2006, 833-840. ACM.
18. Singh S., Lewis R.L., Barto A.G. Where do rewards come from. Proceedings of the annual conference of the cognitive science society 2009, 2601-2606.
19. McDevitt M., Dunn R., Spetch M., Ludvig E. When good news leads to bad choices. J Exp Anal Behav 2016, 105:23-40.
20. Pisklak J.M., McDevitt M.A., Dunn R.M., Spetch M.L. When good pigeons make bad decisions: choice with probabilistic delays and outcomes. J Exp Anal Behav 2015, 104:241-251.
21. Bromberg-Martin E.S., Hikosaka O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 2009, 63:119-126.
22. Bromberg-Martin E.S., Hikosaka O. Lateral habenula neurons signal errors in the prediction of reward information. Nature Neurosci 2011, 14:1209-1216.
23. Blanchard T.C., Hayden B.Y., Bromberg-Martin E.S. Orbitofrontal cortex uses distinct codes for different choice attributes in decisions motivated by curiosity. Neuron 2015, 85:602-614.
24. Singh S., Lewis R.L., Barto A.G., Sorg J. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Trans Autonom Mental Dev 2010, 2:70-82.
25. Guo X., Singh S., Lewis R.L. Reward mapping for transfer in long-lived agents. Adv Neural Inform Process Syst 2013, 2130-2138.
26. Sorg J., Singh S.P., Lewis R.L. Internal rewards mitigate agent boundedness. Proceedings of the 27th international conference on machine learning (ICML-10) 2010, 1007-1014.
27. Hull C. Principles of behavior: an introduction to behavior theory 1943, D. Appleton-Century Company.
28. Keramati M., Gutkin B. Homeostatic reinforcement learning for integrating reward collection and physiological stability. eLife 2014, 3:e04811.
29. Keramati M., Gutkin B.S. A reinforcement learning theory for homeostatic regulation. Advances in neural information processing systems 2011, 82-90.
30. Nahum-Shani I., Smith S.N., Tewari A., Witkiewitz K., Collins L.M., Spring B., Murphy S. Just in time adaptive interventions (JITAIs): an organizing framework for ongoing health behavior support. Methodology Center Technical Report No. 14-126 2014.
31. Nahum-Shani I., Hekler E.B., Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol 2015, 34:1209.
32. Konidaris G., Scheidwasser I., Barto A.G. Transfer in reinforcement learning via shared features. J Mach Learn Res 2012, 13:1333-1371.
33. Gershman S.J., Niv Y. Learning latent structure: carving nature at its joints. Curr Opin Neurobiol 2010, 20:251-256.
34. Daw N. Trial by trial data analysis using computational models. Decision making, affect and learning: attention and performance XXIII 2011, Oxford University Press.
35. Gershman S.J., Norman K.A., Niv Y. Discovering latent causes in reinforcement learning. Curr Opin Behav Sci 2015, 5:43-50.
36. Gershman S.J., Blei D.M., Niv Y. Context, learning, and extinction. Psychol Rev 2010, 117:197.
37. Gershman S.J., Jones C.E., Norman K.A., Monfils M.-H., Niv Y. Gradual extinction prevents the return of fear: implications for the discovery of state. Front Behav Neurosci 2013, 7:164.
38. Gershman S.J., Niv Y. Perceptual estimation obeys Occam's razor. Front Psychol 2013, 4:623.
39. Collins A.G., Brown J.K., Gold J.M., Waltz J.A., Frank M.J. Working memory contributions to reinforcement learning impairments in schizophrenia. J Neurosci 2014, 34:13747-13756.
40. Gershman S.J., Radulescu A., Norman K.A., Niv Y. Statistical computations underlying the dynamics of memory updating. PLoS Computat Biol 2014, 10:e1003939.
41. Niv Y., Daniel R., Geana A., Gershman S.J., Leong Y.C., Radulescu A., Wilson R.C. Reinforcement learning in multidimensional environments relies on attention mechanisms. J Neurosci 2015, 35:8145-8157.
42. Glimcher P.W., Fehr E. Neuroeconomics: decision making and the brain 2013, Academic Press.
43. Levy D.J., Glimcher P.W. The root of all value: a neural common currency for choice. Curr Opin Neurobiol 2012, 22:1027-1038.
44. Padoa-Schioppa C., Assad J.A. Neurons in the orbitofrontal cortex encode economic value. Nature 2006, 441:223-226.
45. Padoa-Schioppa C., Assad J.A. The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nature Neurosci 2008, 11:95-102.
46. Baxter J., Bartlett P.L. Infinite-horizon policy-gradient estimation. J Artif Intell Res 2001, 319-350.
47. Sutton R.S., McAllester D.A., Singh S.P., Mansour Y. Policy gradient methods for reinforcement learning with function approximation. Adv Neural Inform Process Syst 1999, 1057-1063.
48. Bhatnagar S., Ghavamzadeh M., Lee M., Sutton R.S. Incremental natural actor-critic algorithms. Adv Neural Inform Process Syst 2007, 105-112.
49. Li J., Daw N.D. Signals in human striatum are appropriate for policy update rather than value prediction. J Neurosci 2011, 31:5504-5511.
50. Gershman S.J., Moustafa A.A., Ludvig E.A. Time representation in reinforcement learning models of the basal ganglia. Front Computat Neurosci 2014, 7.
51. Daw N.D., Courville A.C., Touretzky D.S. Representation and timing in theories of the dopamine system. Neural Computat 2006, 18:1637-1677.
52. Ludvig E.A., Sutton R.S., Kehoe E.J. Evaluating the TD model of classical conditioning. Learn Behav 2012, 40:305-319.
53. Rivest F., Kalaska J.F., Bengio Y. Alternative time representation in dopamine models. J Computat Neurosci 2010, 28:107-130.
54. Bradtke S.J., Duff M.O. Reinforcement learning methods for continuous time Markov decision problems. Adv Neural Inform Process Syst 1995, 7:393.
55. Takahashi Y., Langdon A.J., Niv Y., Schoenbaum G. Temporal specificity of reward prediction errors signaled by putative dopamine neurons in rat VTA depends on ventral striatum. Neuron 2016.
56. Mello G.B., Soares S., Paton J.J. A scalable population code for time in the striatum. Curr Biol 2015, 25:1113-1122.
57. Gouvêa T.S., Monteiro T., Motiwala A., Soares S., Machens C., Paton J.J. Striatal dynamics explain duration judgments. eLife 2016, 4:e11386.
58. Jin X., Tecuapetla F., Costa R.M. Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences. Nature Neurosci 2014, 17:423-430.
59. Botvinick M.M., Niv Y., Barto A.C. Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 2009, 113:262-280.
60. Botvinick M.M. Hierarchical reinforcement learning and decision making. Curr Opin Neurobiol 2012, 22:956-962.
61. Solway A., Diuk C., Córdova N., Yee D., Barto A.G., Niv Y., Botvinick M.M. Optimal behavioral hierarchy. PLoS Computat Biol 2014, 10. e1003779.
62. Kaelbling L.P., Littman M.L., Cassandra A.R. Planning and acting in partially observable stochastic domains. Artif Intell 1998, 101:99-134.
63. Mnih V., Kavukcuoglu K., Silver D., Rusu A.A., Veness J., Bellemare M.G., Graves A., Riedmiller M., Fidjeland A.K., Ostrovski G. Human-level control through deep reinforcement learning. Nature 2015, 518:529-533.
64. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neurosci 2005, 8:1704-1711.
65. Keramati M., Dezfouli A., Piray P. Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computat Biol 2011, 7. e1002055.
66. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69:1204-1215.
67. Otto A.R., Gershman S.J., Markman A.B., Daw N.D. The curse of planning dissecting multiple reinforcement-learning systems by taxing the central executive. Psychol Sci 2013, 24:751-761.
68. Doll B.B., Duncan K.D., Simon D.A., Shohamy D., Daw N.D. Model-based choices involve prospective neural activity. Nature Neurosci 2015, 18:767-772.
69. Voon V., Derbyshire K., Rück C., Irvine M., Worbe Y., Enander J., Schreiber L., Gillan C., Fineberg N., Sahakian B. Disorders of compulsivity: a common bias towards learning habits. Mol Psychiatry 2015, 20:345-352.
70. Gillan C., Kosinski R.W., Phelps E.A., Daw N.D. Characterizing a psychological dimension related to deficits in goal-directed control. eLife 2016, 5:e11305.
71. Collins A.G., Frank M.J. Opponent actor learning (OpAL): modeling interactive effects of striatal dopamine on reinforcement learning and choice incentive. Psychol Rev 2014, 121:337.
72. Cockburn J., Collins A.G., Frank M.J. A reinforcement learning mechanism responsible for the valuation of free choice. Neuron 2014, 83:551-557.
73. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neurosci 2007, 10:1615-1624.
74. Reynolds J.N., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature 2001, 413:67-70.
75. Reynolds J.N., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 2002, 15:507-521.
76. Morita K., Morishima M., Sakai K., Kawaguchi Y. Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways. Trends Neurosci 2012, 35:457-467.
77. Potjans W., Diesmann M., Morrison A. An imperfect dopaminergic error signal can drive temporal-difference learning. PLoS Computat Biol 2011, 7. e1001133.
78. Potjans W., Morrison A., Diesmann M. A spiking neural network model of an actor-critic learning agent. Neural Computat 2009, 21:301-339.
79. Joel D., Niv Y., Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 2002, 15:535-547.
80. Eshel N., Bukwich M., Rao V., Hemmelder V., Tian J., Uchida N. Arithmetic and local circuitry underlying dopamine prediction errors. Nature 2015, 525:243-246.
81. Rao R.P., Sejnowski T.J. Spike-timing-dependent Hebbian plasticity as temporal difference learning. Neural Computat 2001, 13:2221-2237.
82. Berridge K.C. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 2007, 191:391-431.
83. Redgrave P., Prescott T.J., Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends Neurosci 1999, 22:146-151.
84. Schulz J.M., Redgrave P., Mehring C., Aertsen A., Clements K.M., Wickens J.R., Reynolds J.N. Short-latency activation of striatal spiny neurons via subcortical visual pathways. J Neurosci 2009, 29:6336-6347.
85. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
86. Henny P., Brown M.T., Northrop A., Faunes M., Ungless M.A., Magill P.J., Bolam J.P. Structural correlates of heterogeneous in vivo activity of midbrain dopaminergic neurons. Nature Neurosci 2012, 15:613-619.
87. Brischoux F., Chakraborty S., Brierley D.I., Ungless M.A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A 2009, 106:4894-4899.
88. Joshua M., Adler A., Mitelman R., Vaadia E., Bergman H. Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J Neurosci 2008, 28:11673-11684.
89. Diuk C., Tsai K., Wallis J., Botvinick M., Niv Y. Hierarchical learning induces two simultaneous, but separable, prediction errors in human basal ganglia. J Neurosci 2013, 33:5797-5805.
90. Gershman S.J., Moore C.D., Todd M.T., Norman K.A., Sederberg P.B. The successor representation and temporal context. Neural Computat 2012, 24:1553-1568.
91. Dayan P. Improving generalization for temporal difference learning: the successor representation. Neural Computat 1993, 5:613-624.
92. Matsumoto M., Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 2007, 447:1111-1115.
93. Matsumoto M., Hikosaka O. Representation of negative motivational value in the primate lateral habenula. Nature Neurosci 2009, 12:77-84.
94. Syed E.C., Grima L.L., Magill P.J., Bogacz R., Brown P., Walton M.E. Action initiation shapes mesolimbic dopamine encoding of future rewards. Nature Neurosci 2016, 19:34-36.
95. Phillips P.E., Stuber G.D., Heien M.L., Wightman R.M., Carelli R.M. Subsecond dopamine release promotes cocaine seeking. Nature 2003, 422:614-618.
96. Frank M.J., Seeberger L.C., O'Reilly R.C. By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science 2004, 306:1940-1943.
97. Cox S.M., Frank M.J., Larcher K., Fellows L.K., Clark C.A., Leyton M., Dagher A. Striatal D1 and D2 signaling differentially predict learning from positive and negative outcomes. Neuroimage 2015, 109:95-101.
98. Tai L.-H., Lee A.M., Benavidez N., Bonci A., Wilbrecht L. Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nature Neurosci 2012, 15:1281-1289.
99. Cui G., Jun S.B., Jin X., Pham M.D., Vogel S.S., Lovinger D.M., Costa R.M. Concurrent activation of striatal direct and indirect pathways during action initiation. Nature 2013, 494:238-242.
100. Calabresi P., Picconi B., Tozzi A., Ghiglieri V., Di Filippo M. Direct and indirect pathways of basal ganglia: a critical reappraisal. Nature Neurosci 2014, 17:1022-1030.
101. Balleine B.W. Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol Behav 2005, 86:717-730.
102. Steinberg E.E., Keiflin R., Boivin J.R., Witten I.B., Deisseroth K., Janak P.H. A causal link between prediction errors, dopamine neurons and learning. Nature Neurosci 2013, 16:966-973.
103. Chang C.Y., Esber G.R., Marrero-Garcia Y., Yau H.-J., Bonci A., Schoenbaum G. Brief optogenetic inhibition of dopamine neurons mimics endogenous negative reward prediction errors. Nature Neurosci 2016, 19:111-116.