Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: A simulated robotic study

Neural Networks

Volumn 39, Issue , 2013, Pages 40-51

  Mirolli, Marco ^a   Santucci, Vieri G ^a,b   Baldassarre, Gianluca ^a  

^a INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

^b UNIVERSITY OF PLYMOUTH   (United Kingdom)

Author keywords

Actor critic; Computational model; Intrinsic motivation; Phasic dopamine; Reinforcement learning; TD learning

Indexed keywords

ACTOR CRITIC; COMPUTATIONAL MODEL; INTRINSIC MOTIVATION; PHASIC DOPAMINE; TD-LEARNING;

AMINES; ANIMALS; FORECASTING; REINFORCEMENT LEARNING; ROBOTICS;

NEUROPHYSIOLOGY;

DOPAMINE;

ARTICLE; DOPAMINE RELEASE; ENVIRONMENTAL FACTOR; ERROR; INFORMATION PROCESSING; KINEMATICS; LEARNING; MOTIVATION; PREDICTION; PRIORITY JOURNAL; REINFORCEMENT; REWARD; ROBOTICS; SIMULATION; TASK PERFORMANCE;

ANIMALS; COMPUTER SIMULATION; CORPUS STRIATUM; DOPAMINE; DOPAMINERGIC NEURONS; EXPLORATORY BEHAVIOR; FEEDING BEHAVIOR; HAND STRENGTH; LEARNING; MODELS, NEUROLOGICAL; MOTOR SKILLS; PHOTIC STIMULATION; REINFORCEMENT (PSYCHOLOGY); REWARD; ROBOTICS;

EID: 84872777521     PISSN: 08936080     EISSN: 18792782     Source Type: Journal    
DOI: 10.1016/j.neunet.2012.12.012     Document Type: Article

References (104)

1. Baldassarre G. What are intrinsic motivations? A biological perspective. Proceedings of the international conference on development and learning and epigenetic robotics 2011, E1-E8. IEEE, New York. A. Cangelosi, J. Triesch, I. Fasel, K. Rohlfing, F. Nori, P.Y. Oudeyer, M. Schlesinger, Y. Nagai (Eds.).
2. Intrinsically motivated learning in natural and artificial systems 2013, Springer-Verlag, Berlin. G. Baldassarre, M. Mirolli (Eds.).
3. Barto A. Adaptive critics and the basal ganglia. Models of information processing in the basal ganglia 1995, 215-232. MIT Press, Cambridge, MA. J. Houk, J. Davis, J. Beiser (Eds.).
4. Barto A. Intrinsic motivation and reinforcement learning. Intrinsically motivated learning in natural and artificial systems 2013, Springer-Verlag. G. Baldassarre, M. Mirolli (Eds.).
5. Barto, A., Singh, S., & Chantanez, N. (2004). Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the third international conference on developmental learning. ICDL (pp. 112-119).
6. Barto A., Sutton R., Anderson C. Neuron-like adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics 1983, 13:834-846.
7. Bayer H.M., Glimcher P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 2005, 47:129-141.
8. Berlyne D. Conflict, arousal and curiosity 1960, McGraw Hill, New York.
9. Berridge K. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology 2007, 191:391-431.
10. Bjorklund J., Dunnett S. Dopamine neuron systems in the brain: an update. Trends in Neurosciences 2007, 30:194-202.
11. Butler R.A. Discrimination learning by rhesus monkeys to visual-exploration motivation. Journal of Comparative and Physiology Psychology 1953, 46:95-98.
12. Butler R.A., Harlow H.F. Discrimination learning and learning sets to visual exploration incentives. The Journal of Genetic Psychology 1957, 57:257-264.
13. Calabresi P., Picconi B., Tozzi A., Filippo M.D. Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends in Neuroscience 2007, 30:211-219.
14. Carlton P.L. Scopolamine, amphetamine and light-reinforced responding. Psychonomic Science 1966, 5:347-348.
15. Chiodo L.A., Antelman S.M., Caggiula A.R., Lineberry C.G. Sensory stimuli alter the discharge rate of dopamine (da) neurons: evidence for two functional types of da cells in the substantia nigra. Brain Research 1980, 189:544-549.
16. Csikszentmihalyi M. Flow: the psychology of optimal experience 1991, Harper Perennial, New York, NY.
17. Daw N.D., Niv Y., Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 2005, 8:1704-1711.
18. Dayan P., Sejnowski T. Exploration bonuses and dual control. Machine Learning 1996, 25:5-22.
19. De Charms R. Personal causation: the internal affective determinants of behavior 1968, Academic Press, New York, NY.
20. Dommett E., Coizet V., Blaha C.D., Martindale J., Lefebvre V., Walton N., Mayhew J.E.W., Overton P.G., Redgrave P. How visual stimuli activate dopaminergic neurons at short latency. Science 2005, 307:1476-1479.
21. Doya K. Reinforcement learning in continuous time and space. Neural Computation 2000, 12:219-245.
22. Doya K. Reinforcement learning: computational theory and biological mechanisms. HFSP Journal of Comparative and Physiological Psychology 2007, 1:30-40.
23. Fellous J., Suri R. Roles of dopamine. The handbook of brain theory and neural networks 2003, 361-365. MIT Press, Cambridge, MA. M. Arbib (Ed.).
24. Fiore, V., Mannella, F., Mirolli, M., Gurney, K., & Baldassarre, G. (2008). Instrumental conditioning driven by neutral stimuli: a model tested with simulated robotic rat. In Schlesinger, M., Balkenius, L.B.C. (Eds.) Proceedings of the eight international conference on epigenetic robotics (pp. 13-20).
25. Fiorillo C.D., Newsome W.T., Schultz W. The temporal precision of reward prediction in dopamine neurons. Nature Neuroscience 2008, 11:966-973.
26. Frank M.J. Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated parkinsonism. Journal of Cognitive Neuroscience 2005, 17:51-72.
27. Glimcher P. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences of the United States of America 2011, 108(Suppl 3):15647-15654.
28. Glow P., Winefield A. Response-contingent sensory change in a causally structured environment. Animal Learning & Behavior 1978, 6:1-18. 10.3758/BF03211996.
29. Grace A., Floresco S., Goto Y., Lodge D. Regulation of firing of dopaminergic neuron and control of goal-directed behavior. Trends in Neurosciences 2007, 30:220-227.
30. Grahn J.A., Parkinson J.A., Owen A.M. The role of the basal ganglia in learning and memory: neuropsychological studies. Behavioural Brain Research 2009, 199:53-60.
31. Graybiel A.M. The basal ganglia: learning new tricks and loving it. Current Opinion in Neurobiology 2005, 15:638-644.
32. Graybiel A. Habits, rituals, and the evaluative brain. Annual Review of Neuroscience 2008, 31:359-387.
33. Harlow H.F. Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. Journal of Comparative and Physiological Psychology 1950, 43:289-294.
34. Hollerman J.R., Schultz W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1998, 1:304-309.
35. Horvitz J.C. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 2000, 96:651-656.
36. Horvitz J.C., Stewart T., Jacobs B.L. Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Research 1997, 759:251-258.
37. Houk J., Adams J., Barto A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of information processing in the basal ganglia 1995, 249-270. MIT Press, Cambridge, MA. J. Houk, J. Davis, D. Beiser (Eds.).
38. Huang X., Weng J. Novelty and reinforcement learning in the value system of developmental robots. Proceedings of the second international workshop epigenetic robotics: modeling cognitive development in robotic systems 2002, 47-55. Lund University Cognitive Studies, Lund. C. Prince, Y. Demiris, Y. Marom, H. Kozima, C. Balkenius (Eds.).
39. Hull C.L. Principles of behavior 1943, Appleton-Century-Crofts.
40. Joel D., Niv Y., Ruppin E. Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Networks 2002, 15:535-547.
41. Kakade S., Dayan P. Dopamine: generalization and bonuses. Neural Networks 2002, 15:549-559.
42. Kaplan F., Oudeyer P. Motivational principles for visual know-how development. Proceedings of the third international workshop on epigenetic robotics 2003, 73-80. Lund University Cognitive Studies, Lund. C. Prince, L. Berthouze, H. Kozima, D. Bullock, G. Stojanov, C. Balkenius (Eds.).
43. Khamassi M., Lacheze L., Girard B., Berthoz A., Guillot A. Actor-critic models of reinforcement learning in the basal ganglia: from natural to artificial rats. Adaptive Behavior 2005, 13:131-148.
44. Kish G.B. Learning when the onset of illumination is used as reinforcing stimulus. Journal of Comparative and Physiological Psychology 1955, 48:261-264.
45. Lee R., Walker R., Meeden L., Marshall J. Category-based intrinsic motivations. Proceedings og the ninth international conference on epigenetic robotics 2009, 81-88. Lund University Cognitive Studies, Lund. L. Canamero, P. Oudeyer, C. Balkenius (Eds.).
46. Ljungberg T., Apicella P., Schultz W. Responses of monkey midbrain dopamine neurons during delayed alternation performance. Brain Research 1991, 567:337-341.
47. Ljungberg T., Apicella P., Schultz W. Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology 1992, 67:145-163.
48. May P.J., McHaffie J.G., Stanford T.R., Jiang H., Costello M.G., Coizet V., Hayes L.M., Haber S.N., Redgrave P. Tectonigral projections in the primate: a pathway for pre-attentive sensory input to midbrain dopaminergic neurons. European Journal of Neuroscience 2009, 29:575-587.
49. Mirolli M., Baldassarre G. Functions and mechanisms of intrinsic motivations: the knowledge vs. competence distinction. Intrinsically motivated learning in natural and artificial systems 2013, Springer-Verlag, Berlin. G. Baldassarre, M. Mirolli (Eds.).
50. Montague P.R., Hyman S.E., Cohen J.D. Computational roles for dopamine in behavioural control. Nature 2004, 431:760-767.
51. Montgomery K. The role of the exploratory drive in learning. Journal of Comparative Psychology 1954, 47:60-64.
52. Morris G., Arkadir D., Nevet A., Vaadia E., Bergman H. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 2004, 43:133-143.
53. Ng A.Y., Harada D., Russell S. Policy invariance under reward transformations: theory and application to reward shaping. Proceedings of the 16th international conference of machine learning 1999, 278-287. Morgan Kaufmann Publisher Inc., San Francisco, CA, USA.
54. Oudeyer P., Kaplan F., Hafner V. Intrinsic motivation system for autonomous mental development. IEEE Transactions on Evolutionary Computation 2007, 703-713.
55. Pouget A., Snyder L.H. Nature Neuroscience 2000, 3(Suppl):1192-1198.
56. Redgrave P., Gurney K. The short-latency dopamine signal: a role in discovering novel actions?. Nature Reviews Neuroscience 2006, 7:967-975.
57. Redgrave P., Gurney K., Reynolds J. What is reinforced by phasic dopamine signals?. Brain Research Reviews 2008, 58:322-339.
58. Redgrave P., Gurney K., Stafford T., Thirkettle M., Lewis J. The role of the basal ganglia in discovering novel actions. Intrinsically motivated learning in natural and artificial systems 2013, Springer-Verlag. G. Baldassarre, M. Mirolli (Eds.).
59. Redgrave P., Prescott T.J., Gurney K. Is the short-latency dopamine response too short to signal reward error?. Trends in Neuroscience 1999, 22:146-151.
60. Redgrave P., Vautrelle N., Reynolds J.N.J. Functional properties of the basal ganglia's re-entrant loop architecture: selection and reinforcement. Neuroscience 2011.
61. Reed P., Mitchell C., Nokes T. Intrinsic reinforcing properties of putatively neutral stimuli in an instrumental two-lever discrimination task. Animal Learning and Behavior 1996, 24:38-45.
62. Reynolds J.N., Hyland B.I., Wickens J.R. A cellular mechanism of reward-related learning. Nature 2001, 413:67-70.
63. Reynolds J.N.J., Wickens J.R. Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks 2002, 15:507-521.
64. Robbins T., Everitt B. Functions of dopamine in the dorsal and ventral striatum. Seminars in Neuroscience 1992, 4:119-128.
65. Roberts C.L., Marx M.H., Collier G. Light onset and light offset as reinforcers for the albino rat. Journal of Comparative and Physiological Psychology 1958, 51:575-579.
66. Robinson S., Sotak B., During M., Palmiter R. Local dopamine production in the dorsal striatum restores goal-directed behavior in dopamine-deficient mice. Behavioral Neuroscience 2006, 120:196-200.
67. Romanelli P., Esposito V., Schaal D.W., Heit G. Somatotopy in the basal ganglia: experimental and clinical evidence for segregated sensorimotor channels. Brain Research Reviews 2005, 48:112-128.
68. Romo R., Schultz W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. Journal of Neurophysiology 1990, 63:592-606.
69. Russell A., Glow P. Some effects of short-term immediate prior exposure to light change on responding for light change. Animal Learning & Behavior 1974, 2:262-266. 10.3758/BF03199191.
70. Ryan, Deci Intrinsic and extrinsic motivations: classic definitions and new directions. Contemporary Educational Psychology 2000, 25:54-67.
71. Salzman C.D., Belova M.A., Paton J.J. Beetles, boxes and brain cells: neural mechanisms underlying valuation and learning. Current Opinion in Neurobiology 2005, 15:721-729.
72. Santucci, V., Baldassarre, G., & Mirolli, M. (2012). Intrinsic motivation mechanisms for competence acquisition. In Proceedings of ICDL-Epirob 2012, San Diego.
73. Schembri M., Mirolli M., Baldassarre G. Evolution and learning in an intrinsically motivated reinforcement learning robot. Advances in artificial life 2007, 294-333. Springer, Berlin. F.A. y Costa, L. Rocha, E. Costa, I. Harvey, A. Coutinho (Eds.).
74. Schembri M., Mirolli M., Baldassarre G. Evolving childhood's length and learning parameters in an intrinsically motivated reinforcement learning robot. Proceedings of the seventh international conference on epigenetic robotics 2007, 141-148. Lund University Cognitive Studies, Lund. L. Berthouze, G. Dhristiopher, M. Littman, H. Kozima, C. Balkenius (Eds.).
75. Schembri M., Mirolli M., Baldassarre G. Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. Proceedings of the 6th international conference on development and learning 2007, E1-E6. Imperial College, London. Y. Demiris, D. Mareschal, B. Scassellati, J. Weng (Eds.).
76. Schmidhuber J. A possibility for implementing curiosity and boredom in model-building neural controllers. Proceedings of the international conference on simulation of adaptive behavior: from animals to animats 1991, 222-227. MIT Press/Bradford Books, Cambridge, Massachusetts/London, England. J. Meyer, S. Wilson (Eds.).
77. Schmidhuber J. Curious model-building control system. Proceedings of international joint conference on neural networks 1991, 1458-1463. IEEE, Singapore.
78. Schultz W. Predictive reward signal of dopamine neurons. Journal of Neurophysiology 1998, 80:1-27.
79. Schultz W. Getting formal with dopamine and reward. Neuron 2002, 36:241-263.
80. Schultz W. Behavioral theories and the neurophysiology of reward. Annual Review of Psychology 2006, 57:87-115.
81. Schultz W. Multiple dopamine functions at different time scales. Annual Review of Neuroscience 2007, 30:259-288.
82. Schultz W., Apicella P., Ljumberg T. Responses of monkey dopamine neurons to reaward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience 1993, 13:900-913.
83. Schultz W., Dayan P., Montague P.R. A neural substrate of prediction and reward. Science 1997, 275:1593-1599.
84. Singh S., Barto R.L.A., Sorg J. Instrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Transactions on Autonomous Mental Development 2010, 2:70-82.
85. Steinfels G.F., Heym J., Strecker R.E., Jacobs B.L. Response of dopaminergic neurons in cat to auditory stimuli presented across the sleep-waking cycle. Brain Research 1983, 277:150-154.
86. Stout, A., & Barto, A.G. (2010). Competence progress intrinsic motivation. In Proceedings of the international conference on development and learning. ICDL (pp. 257-262).
87. Strecker R.E., Jacobs B.L. Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Research 1985, 361:339-350.
88. Sugrue L.P., Corrado G.S., Newsome W.T. Choosing the greater of two goods: neural currencies for valuation and decision making. Nature Review Neuroscience 2005, 6:363-375.
89. Suri R.E. Td models of reward predictive responses in dopamine neurons. Neural Networks 2002, 15:523-533.
90. Sutton R. Learning to predict by the methods of temporal differences. Machine Learning 1988, 3:9-44.
91. Sutton R. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the seventh international conference on machine learning 1990, 216-224. Morgan Kaufmann.
92. Sutton R., Barto A. Reinforcement learning: an introduction 1998, MIT Press, Cambridge, MA.
93. Sutton R., Precup D., Singh S. Between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. Artificial Intelligence 1999, 112:181-211.
94. Sutton R., Tanner B. Temporal-difference networks. Advances in neural information processing systems 2005, 17:1377-1384.
95. Tobler P.N., Fiorillo C.D., Schultz W. Adaptive coding of reward value by dopamine neurons. Science 2005, 307:1642-1645.
96. Uchibe E., Doya K. Finding intrinsic rewards by embodied evolution and constrained reinforcement learning. Neural Networks 2008, 21:1447-1455.
97. Ungless M. Dopamine: the salient issue. Trends in Neuroscience 2004, 27:702-706.
98. Waelti P., Dickinson A., Schultz W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 2001, 412:43-48.
99. White R. Motivation reconsidered: the concept of competence. Psychological Review 1959, 66:297-333.
100. Wickens J.R. Synaptic plasticity in the basal ganglia. Behavioural Brain Research 2009, 199:119-128.
101. Williams D., Lowe G. Response contingent illumination change as a reinforcer in the rat. Animal Behaviour 1972, 20:259-262.
102. Wise R. Dopamine, learning and motivation. Nature Review Neuroscience 2004, 5:483-494.
103. Wise R., Rompre P. Brain dopamine and reward. Annual Review of Psychology 1989, 40:191-225.
104. Zweifel L.S., Parker J.G., Lobb C.J., Rainwater A., Wall V.Z., Fadok J.P., Darvas M., Kim M.J., Mizumori S.J.Y., Paladini C.A., Phillips P.E.M., Palmiter R.D. Disruption of nmdar-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proceedings of the National Academy of Sciences of the United States of America 2009, 106:7281-7288.