Learning to represent reward structure: A key to adapting to complex environments

Neuroscience Research

Volumn 74, Issue 3-4, 2012, Pages 177-183

  Nakahara, Hiroyuki ^a   Hikosaka, Okihide ^b  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

^b NATIONAL EYE INSTITUTE   (United States)

Author keywords

Decision; Dopamine; Reinforcement learning; Reward; Salience; Structure; Value

Indexed keywords

DOPAMINE;

ARTICLE; ASSOCIATION; DOPAMINERGIC ACTIVITY; LEARNING; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REWARD;

ANIMALS; BRAIN; DOPAMINE; HUMANS; LEARNING; MODELS, NEUROLOGICAL; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 84871931160     PISSN: 01680102     EISSN: 18728111     Source Type: Journal    
DOI: 10.1016/j.neures.2012.09.007     Document Type: Article

References (78)

1. Acuna D.E., Schrater P. Structure learning in human sequential decision-making. PLoS Comput. Biol. 2010, 6(12):e1001003.
2. Balleine B.W., Daw N., O'Doherty J.P. Multiple Forms of Value Learning and the Function of Dopamine, Neuroeconomics Decision Making and the Brain 2008, Elsevier, Amsterdam, pp. 367-387.
3. Bao S., Chan V.T., Merzenich M.M. Cortical remodelling induced by activity of ventral tegmental dopamine neurons. Nature 2001, 412(6842):79-83.
4. Bapi R.S., Miyapuram K.P., Graydon F.X., Doya K. fMRI investigation of cortical and subcortical networks in the learning of abstract and effector-specific representations of motor sequences. NeuroImage 2006, 32(2):714-727.
5. Barto A. Adaptive critics and the basal ganglia. Models of Information Processing in the Basal Ganglia 1994, 12-31. The MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
6. Bayer H., Glimcher P. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47(1):129-141.
7. Brischoux F., Chakraborty S., Brierley D.I., Ungless M.A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc. Natl. Acad. Sci. U.S.A. 2009, 106(12):4894-4899.
8. Bromberg-Martin E.S., Hikosaka O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 2009, 63(1):119-126.
9. Bromberg-Martin E.S., Matsumoto M., Hikosaka O. Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron 2010, 67(1):144-155.
10. Bromberg-Martin E.S., Matsumoto M., Nakahara H., Hikosaka O. Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 2010, 67(3):499-510.
11. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482(7383):85-88.
12. Costa R.M. A selectionist account of de novo action learning. Curr. Opin. Neurobiol. 2011, 21(4):579-586.
13. Courville A., Daw N., Touretzky D. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 2006, 10(7):294-300.
14. Daw N.D., Courville A.C., Touretzky D.S. Representation and timing in theories of the dopamine system. Neural Comput. 2006, 18(7):1637-1677.
15. Daw N.D., Gershman S.J., Seymour B., Dayan P., Dolan R.J. Model-based influences on humans' choices and striatal prediction errors. Neuron 2011, 69(6):1204-1215.
16. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 2005, 8(12):1704-1711.
17. Dayan P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 1993, 5:613-624.
18. Dayan P.G., Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr. Opin. Neurobiol. 2008, 18(2):185-196.
19. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?. Neural Netw. 1999, 12(7-8):961-974.
20. Doya K. Reinforcement learning: computational theory and biological mechanisms. HFSP J. 2007, 1(1):30-40.
21. Enomoto K., Matsumoto N., Nakai S., Satoh T., Sato T.K., Ueda Y., Inokawa H., Haruno M., Kimura M. Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc. Natl. Acad. Sci. U.S.A. 2011, 108(37):15462-15467.
22. Fiorillo C.D., Tobler P.N., Schultz J. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 2003, 299:1898-1902.
23. Frank M.J., Surmeier D.J. Do substantia nigra dopaminergic neurons differentiate between reward and punishment?. J. Mol. Cell Biol. 2009, 1(1):15-16.
24. Funamizu A., Ito M., Doya K., Kanzaki R., Takahashi H. Uncertainty in action-value estimation affects both action choice and learning rate of the choice behaviors of rats. Eur. J. Neurosci. 2012, 35(7):1180-1189.
25. Gershman S.J., Moore C.D., Todd M.T., Norman K.N., Sederberg P.B. The successor representation and temporal context. Neural Comput. 2012, 24:1-16.
26. Gershman S.J., Niv Y. Learning latent structure: carving nature at its joints. Curr. Opin. Neurobiol. 2010, 20(2):251-256.
27. Gershman S.J., Pesaran B., Daw N.D. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. J. Neurosci. 2009, 29(43):13524-13531.
28. Gläscher J., Daw N., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66(4):585-595.
29. Glimcher P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc. Natl. Acad. Sci. U.S.A. 2011, 108(Suppl. 3):15647-15654.
30. Green C.S., Benson C., Kersten D., Schrater P. Alterations in choice behavior by manipulations of world model. Proc. Natl. Acad. Sci. U.S.A. 2010, 107(37):16401-16406.
31. Gruber A.J., Dayan P., Gutkin B.S., Solla S.A. Dopamine modulation in the basal ganglia locks the gate to working memory. J. Comput. Neurosci. 2006, 20(2):153-166.
32. Hayden B.Y., Heilbronner S., Pearson J., Platt M.L. Neurons in anterior cingulate cortex multiplex information about reward and action. J. Neurosci. 2010, 30(9):3339-3346.
33. Hayden B.Y., Pearson J.M., Platt M.L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat. Neurosci. 2011, 14(7):933-939.
34. Hikosaka O., Nakahara H., Rand M.K., Sakai K., Lu X., Nakamura K., Miyachi S., Doya K. Parallel neural networks for learning sequential procedures. Trends Neurosci. 1999, 22(10):464-471.
35. Hikosaka O., Nakamura K., Nakahara H. Basal ganglia orient eyes to reward. J. Neurophysiol. 2006, 95(2):567-584.
36. Houk J.C., Adams J.L., Barto A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia 1994, 249-252. The MIT Press, Cambridge, MA. J.C. Houk, J.L. Davis, D.G. Beiser (Eds.).
37. Joshua M., Adler A., Rosin B., Vaadia E., Bergman H. Encoding of probabilistic rewarding and aversive events by pallidal and nigral neurons. J. Neurophysiol. 2009, 101(2):758-772.
38. Kakade S., Dayan P. Dopamine: generalization and bonuses. Neural Netw. 2002, 15(4-6):549-559.
39. Kolling N., Behrens T.E., Mars R.B., Rushworth M.F. Neural mechanisms of foraging. Science 2012, 336(6077):95-98.
40. Lammel S., Hetzel A., Häckel O., Jones I., Liss B., Roeper J. Unique properties of mesoprefrontal neurons within a dual mesocorticolimbic dopamine system. Neuron 2008, 57(5):760-773.
41. Ludvig E.A., Sutton R.S., Kehoe E.J. Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput. 2008, 20(12):3034-3054.
42. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459(7248):837-841.
43. McDannald M.A., Lucantonio F., Burke K.A., Niv Y., Schoenbaum G. Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. J. Neurosci. 2011, 31(7):2700-2705.
44. McDannald M.A., Takahashi Y.K., Lopatina N., Pietras B.W., Jones J.L., Schoenbaum G. Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur. J. Neurosci. 2012, 35(7):991-996.
45. Montague P., Dayan P., Sejnowski T. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 1996, 16(5):1936-1947.
46. Montague P.R., Hyman S.E., Cohen J.D. Computational roles for dopamine in behavioural control. Nature 2004, 431(7010):760-767.
47. Montague P.R., King-Casas B., Cohen J.D. Imaging valuation models in human choice. Annu. Rev. Neurosci. 2006, 29:417-448.
48. Morris G., Nevet A., Arkadir D., Vaadia E., Bergman H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 2006, 9(8):1057-1063.
49. Nakahara H., Doya K., Hikosaka O. Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences - a computational approach. J. Cogn. Neurosci. 2001, 13(5):626-647.
50. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 2004, 41:269-280.
51. Nakahara H., Kaveri S. Internal-time temporal difference model for neural value-based decision making. Neural Comput. 2010, 22(12):3062-3106.
52. Nomoto K., Schultz W., Watanabe T., Sakagami M. Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J. Neurosci. 2010, 30(32):10692-10702.
53. Palminteri S., Boraud T., Lafargue G., Dubois B., Pessiglione M. Brain hemispheres selectively track the expected value of contralateral options. J. Neurosci. 2009, 29(43):13465-13472.
54. Parr, R., Painter-Wakefield, C., Li, L., Littman, M., 2007. Analyzing feature generation for value-function approximation. In: ICML-07, Oregon, USA, pp.737-744.
55. Rangel A., Camerer C., Montague P.R. A framework for studying the neurobiology of value-based decision making. Nat. Rev. Neurosci. 2008, 9(7):545-556.
56. Rao R.P. Decision making under uncertainty: a neural model based on partially observable markov decision processes. Front. Comput. Neurosci. 2010, 4:146.
57. Redgrave P., Gurney K. The short-latency dopamine signal: a role in discovering novel actions?. Nat. Rev. Neurosci. 2006, 7(12):967-975.
58. Redish A.D., Jensen S., Johnson A., Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol. Rev. 2007, 114(3):784-805.
59. Reed A., Riley J., Carraway R., Carrasco A., Perez C., Jakkamsetti V., Kilgard M.P. Cortical map plasticity improves learning but is not necessary for improved performance. Neuron 2011, 70(1):121-131.
60. Reynolds J.N., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw. 2002, 15(4-6):507-521.
61. Ribas-Fernandes J.J.F., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., Botvinick M.M. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71(2):370-379.
62. Roesch M.R., Calu D.J., Esber G.R., Schoenbaum G. Neural correlates of variations in event processing during learning in basolateral amygdala. J. Neurosci. 2010, 30(7):2464-2471.
63. Roesch M.R., Calu D.J., Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat. Neurosci. 2007, 10(12):1615-1624.
64. Rushworth M.F., Kolling N., Sallet J., Mars R.B. Valuation and decision-making in frontal cortex: one or many serial or parallel systems?. Curr. Opin. Neurobiol. 2012, (Epub ahead of print).
65. Satoh T., Nakai S., Sato T., Kimura M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 2003, 23(30):9913-9923.
66. Schultz W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 1998, 80:1-27.
67. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275(5306):1593-1599.
68. Seitz A.R., Dinse H.R. A common framework for perceptual learning. Curr. Opin. Neurobiol. 2007, 17(2):148-153.
69. Singh S., Barto A.G., Chentanez N. Intrinsically Motivated Reinforcement Learning. Advances in Neural Information Processing Systems (NIPS) 2005, 17:1281-1288.
70. Sutton R., Barto A.G. Reinforcement Learning: An Introduction 1998, The MIT Press, Cambridge, MA.
71. Sutton R.S., Barto A.G. Time-derivative models of pavlovian reinforcement. Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990, 497-537. The MIT Press, Cambridge, MA. M. Gabriel, J. Moore (Eds.).
72. Sutton R.S., Maei H.R., Precup D., Bhatnagar S., Silver D., Szepesvari C., Wiewiora E. Fast gradient-descent methods for temporal-difference learning with linear function approximation. ICML-09 2009, 993-1000.
73. Sutton R.S., Modayil J., Delp M., Degris T., Pilarski P.M., White A. Horde: A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction. AAMAS'11 The 10th International Conference on Autonomous Agents and Multiagent Systems - vol. 2 2011, 761-768.
74. Suzuki S., Harasawa N., Ueno K., Gardner J.L., Ichinohe N., Haruno M., Cheng K., Nakahara H. Learning to simulate others' decisions. Neuron 2012, 74:1125-1137.
75. Todd M., Niv Y., Cohen J.D. Learning to use working memory in partially observable environments through dopaminergic reinforcement. Advances in Neural Information Processing Systems (NIPS) 2009, 21:1-8.
76. Wan X., Nakatani H., Ueno K., Asamizuya T., Cheng K., Tanaka K. The neural basis of intuitive best next-move generation in board game experts. Science 2011, 331(6015):341-346.
77. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat. Neurosci. 2012, 1-19.
78. Zacks J.M., Kurby C.A., Eisenberg M.L., Haroutunian N. Prediction error associated with the perceptual segmentation of naturalistic events. J. Cogn. Neurosci. 2011, 23(12):4057-4066.