Multiplexing signals in reinforcement learning with internal models and dopamine

Current Opinion in Neurobiology

Volumn 25, Issue , 2014, Pages 123-129

  Hiroyuki, Nakahara ^a  

^a RIKEN BRAIN SCIENCE INSTITUTE   (Japan)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE;

COGNITION; DECISION MAKING; FEEDBACK SYSTEM; FORAGING; MESENCEPHALON; PERCEPTION; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REVIEW; REWARD; SIGNAL TRANSDUCTION; ANIMAL; BRAIN; HUMAN; METABOLISM; PHYSIOLOGY;

ANIMALS; BRAIN; DECISION MAKING; DOPAMINE; HUMANS; REINFORCEMENT (PSYCHOLOGY); REWARD;

EID: 84892891610     PISSN: 09594388     EISSN: 18736882     Source Type: Journal    
DOI: 10.1016/j.conb.2014.01.001     Document Type: Review

References (78)

1. Dayan P., Niv Y. Reinforcement learning: the good, the bad and the ugly. Curr Opin Neurobiol 2008, 18:185-196.
2. Gershman S.J., Niv Y. Exploring a latent cause theory of classical conditioning. Learn Behav 2012, 40:255-268.
3. Doll B.B., Simon D.A., Daw N.D. The ubiquity of model-based reinforcement learning. Curr Opin Neurobiol 2012, 22:1-7.
4. McDannald M.A., Takahashi Y.K., Lopatina N., Pietras B.W., Jones J.L., Schoenbaum G. Model-based learning and the contribution of the orbitofrontal cortex to the model-free world. Eur J Neurosci 2012, 35:991-996.
5. Redish A.D., Jensen S., Johnson A., Kurth-Nelson Z. Reconciling reinforcement learning models with behavioral extinction and renewal: implications for addiction, relapse, and problem gambling. Psychol Rev 2007, 114:784-805.
6. Hazy T.E., Frank M.J., O'Reilly R.C. Neural mechanisms of acquired phasic dopamine responses in learning. Neurosci Biobehav Rev 2010, 34:701-720.
7. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
8. Montague P., Dayan P., Sejnowski T. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 1996, 16:1936-1947.
9. Nakahara H., Kaveri S. Internal-time temporal difference model for neural value-based decisionmaking. Neural Comput 2010, 22:3062-3106.
10. Schultz W., Apicella P., Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 1993, 13:900-913.
11. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 2005, 8:1704-1711.
12. Green C.S., Benson C., Kersten D., Schrater P. Alterations in choice behavior by manipulations of world model. Proc Natl Acad Sci U S A 2010, 107:16401-16406.
13. Wunderlich K., Dayan P., Dolan R.J. Mapping value based planning and extensively trained choice in the human brain. Nat Neurosci 2012, 1-19.
14. Gläscher J., Daw N.D., Dayan P., O'Doherty J.P. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 2010, 66:585-595.
15. Ito M., Doya K. Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit. Curr Opin Neurobiol 2011, 21:1-6.
16. Hayden B.Y., Pearson J.M., Platt M.L. Neuronal basis of sequential foraging decisions in a patchy environment. Nat Neurosci 2011, 14:933-939.
17. Kolling N., Behrens T.E., Mars R.B., Rushworth M.F. Neural mechanisms of foraging. Science 2012, 336:95-98.
18. Adams G.K., Watson K.K., Pearson J., Platt M.L. Neuroethology of decision-making. Curr Opin Neurobiol 2012, 22:982-989.
19. Bernacchia A., Seo H., Lee D., Wang X.-J. A reservoir of time constants for memory traces in cortical neurons. Nat Neurosci 2011, 14:366-372.
20. Lohrenz T., McCabe K., Camerer C.F., Montague P.R. Neural signature of fictive learning signals in a sequential investment task. Proc Natl Acad Sci U S A 2007, 104:9493-9498.
21. Boorman E.D., Behrens T.E., Rushworth M.F. Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol 2011, 9.
22. Ribas-Fernandes J.J.F., Solway A., Diuk C., McGuire J.T., Barto A.G., Niv Y., Botvinick M.M. A neural signature of hierarchical reinforcement learning. Neuron 2011, 71:370-379.
23. Preuschoff K., Quartz S.R., Bossaerts P. Human insula activation reflects risk prediction errors as well as risk. J Neurosci 2008, 28:2745-2752.
24. Behrens T.E., Hunt L.T., Rushworth M.F. The computation of social behavior. Science 2009, 324:1160-1164.
25. Dunne S., O'Doherty J.P. Insights from the application of computational neuroimaging to social neuroscience. Curr Opin Neurobiol 2013, 23:1-6.
26. Rilling J.K., King-Casas B., Sanfey A.G. The neurobiology of social decision-making. Curr Opin Neurobiol 2008, 18:159-165.
27. Seo H., Lee D. Neural basis of learning and preference during social decision-making. Curr Opin Neurobiol 2012, 22:990-995.
28. Yoshida K., Saito N., Iriki A., Isoda M. Social error monitoring in macaque frontal cortex. Nat Neurosci 2012, 15:1307-1312.
29. Burke C.J., Tobler P.N., Baddeley M., Schultz W. Neural mechanisms of observational learning. Proc Natl Acad Sci U S A 2010, 107:14431-14436.
30. Behrens T.E., Hunt L.T., Woolrich M.W., Rushworth M.F. Associative learning of social value. Nature 2008, 456:245-249.
31. Yoshida W., Seymour B., Friston K.J., Dolan R.J. Neural mechanisms of belief inference during cooperative games. J Neurosci 2010, 30:10744-10751.
32. Hampton A.N., Bossaerts P., O'Doherty J.P. Neural correlates of mentalizing-related computations during strategic interactions in humans. Proc Natl Acad Sci U S A 2008, 105:6741-6746.
33. Coricelli G., Nagel R. Neural correlates of depth of strategic reasoning in medial prefrontal cortex. Proc Natl Acad Sci U S A 2009, 106:9163-9168.
34. Haruno M., Frith C.D. Activity in the amygdala elicited by unfair divisions predicts social value orientation. Nat Neurosci 2009, 13:160-161.
35. Xiang T., Ray D., Lohrenz T., Dayan P., Montague P.R. Computational phenotyping of two-person interactions reveals differential neural response to depth-of-thought. PLoS Comput Biol 2012, 8:e1002841.
36. Suzuki S., Harasawa N., Ueno K., Gardner J.L., Ichinohe N., Haruno M., Cheng K., Nakahara H. Learning to simulate others' decisions. Neuron 2012, 74:1125-1137.
37. Enomoto K., Matsumoto N., Nakai S., Satoh T., Sato T.K., Ueda Y., Inokawa H., Haruno M., Kimura M. Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc Natl Acad Sci U S A 2011, 108:15462-15467.
38. Cohen J.Y., Haesler S., Vong L., Lowell B.B., Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 2012, 482:85-88.
39. Steinberg E.E., Keiflin R., Boivin J.R., Witten I.B., Deisseroth K., Janak P.H. A causal link between prediction errors, dopamine neurons and learning. Nat Neurosci 2013, 16:966-973.
40. Schultz W. Updating dopamine reward signals. Curr Opin Neurobiol 2013, 23:229-238.
41. Glimcher P.W. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proc Natl Acad Sci U S A 2011, 108:15647-15654.
42. Nomoto K., Schultz W., Watanabe T., Sakagami M. Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J Neurosci 2010, 30:10692-10702.
43. Fiorillo C.D., Song M.R., Yun S.R. Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli. J Neurosci 2013, 33:4710-4725.
44. Fiorillo C.D., Yun S.R., Song M.R. Diversity and homogeneity in responses of midbrain dopamine neurons. J Neurosci 2013, 33:4693-4709.
45. Bromberg-Martin E.S., Hikosaka O. Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 2009, 63:119-126.
46. Howe M.W., Tierney P.L., Sandberg S.G., Phillips P.E., Graybiel A.M. Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 2013, 500:575-579.
47. Flagel S.B., Clark J.J., Robinson T.E., Mayo L., Czuj A., Willuhn I., Akers C.A., Clinton S.M., Phillips P.E., Akil H. A selective role for dopamine in stimulus-reward learning. Nature 2011, 469:53-57.
48. Matsumoto M., Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 2009, 459:837-841.
49. de Lafuente V., Romo R. Dopamine neurons code subjective sensory experience and uncertainty of perceptual decisions. Proc Natl Acad Sci U S A 2011, 108:19767-19771.
50. Bromberg-Martin E.S., Matsumoto M., Hong S., Hikosaka O. A pallidus-habenula-dopamine pathway signals inferred stimulus values. J Neurophysiol 2010, 104:1068-1076.
51. Gan J.O., Walton M.E., Phillips P.E. Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat Neurosci 2010, 13:25-27.
52. Jin X., Costa R.M. Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature 2010, 466:457-462.
53. Nakahara H., Itoh H., Kawagoe R., Takikawa Y., Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron 2004, 41:269-280.
54. Takahashi Y.K., Roesch M.R., Wilson R.C., Toreson K., O'Donnell P., Niv Y., Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex. Nat Neurosci 2011.
55. Bromberg-Martin E.S., Matsumoto M., Nakahara H., Hikosaka O. Multiple timescales of memory in lateral habenula and dopamine neurons. Neuron 2010, 67:499-510.
56. Nakamura K., Santos G.S., Matsuzaki R., Nakahara H. Differential reward coding in the subdivisions of the primate caudate during an oculomotor task. J Neurosci 2012, 32:15963-15982.
57. Nakahara H., Hikosaka O. Learning to represent reward structure: a key to adapting to complex environments. Neurosci Res 2012, 74:177-183.
58. Dayan P. How to set the switches on this thing. Curr Opin Neurobiol 2012.
59. Matsumoto M., Takada M. Distinct representations of cognitive and motivational signals in midbrain dopamine neurons. Neuron 2013, 79:1011-1024.
60. Bromberg-Martin E.S., Matsumoto M., Hikosaka O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 2010, 68:815-834.
61. Fiorillo C.D. Two dimensions of value: dopamine neurons represent reward but not aversiveness. Science 2013, 341:546-549.
62. Lammel S., Lim B.K., Ran C., Huang K.W., Betley M.J., Tye K.M., Deisseroth K., Malenka R.C. Input-specific control of reward and aversion in the ventral tegmental area. Nature 2012.
63. Montague P.R., Dolan R.J., Friston K.J., Dayan P. Computational psychiatry. Trends Cogn Sci 2012, 16:72-80.
64. Hikosaka O., Bromberg-Martin E., Hong S., Matsumoto M. New insights on the subcortical representation of reward. Curr Opin Neurobiol 2008, 18:203-208.
65. Klein-Flugge M.C., Hunt L.T., Bach D.R., Dolan R.J., Behrens T.E. Dissociable reward and timing signals in human midbrain and ventral striatum. Neuron 2011, 72:654-664.
66. Fujisawa S., Buzsaki G. A 4Hz oscillation adaptively synchronizes prefrontal, VTA, and hippocampal activities. Neuron 2011, 72:153-165.
67. van der Meer M.A.A., Johnson A., Schmitzer-Torbert N.C., Redish A.D. Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task. Neuron 2010, 67:25-32.
68. Wimmer G.E., Shohamy D. Preference by association: how memory mechanisms in the hippocampus bias decisions. Science 2012, 338:270-273.
69. Aizawa H., Yanagihara S., Kobayashi M., Niisato K., Takekawa T., Harukuni R., McHugh T.J., Fukai T., Isomura Y., Okamoto H. The synchronous activity of lateral habenular neurons is essential for regulating hippocampal theta oscillation. J Neurosci 2013, 33:8909-8921.
70. Kennerley S.W., Behrens T.E., Wallis J.D. Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat Neurosci 2011, 14:1581-1589.
71. Lee D., Seo H., Jung M.W. Neural basis of reinforcement learning and decision making. Ann Rev Neurosci 2012.
72. Rushworth M.F., Noonan M.P., Boorman E.D., Walton M.E., Behrens T.E. Frontal cortex and reward-guided learning and decision-making. Neuron 2011, 70:1054-1069.
73. Kable J.W., Glimcher P.W. The neurobiology of decision: consensus and controversy. Neuron 2009, 63:733-745.
74. Hare T.A., Camerer C.F., Rangel A. Self-control in decision-making involves modulation of the vmPFC valuation system. Science 2009, 324:646-648.
75. Wallis J.D. Cross-species studies of orbitofrontal cortex and value-based decision-making. Nat Neurosci 2012, 15:13-19.
76. Chang S.W., Brent L.J., Adams G.K., Klein J.T., Pearson J.M., Watson K.K., Platt M.L. Neuroethology of primate social behavior. Proc Natl Acad Sci U S A 2013, 110:10387-10394.
77. Carter R.M., Bowling D.L., Reeck C., Huettel S.A. A distinct role of the temporal-parietal junction in predicting socially guided decisions. Science 2012, 337:109-111.
78. Pearson J.M., Heilbronner S.R., Barack D.L., Hayden B.Y., Platt M.L. Posterior cingulate cortex: adapting behavior to a changing world. Trends Cogn Sci 2011, 15:143-151.