In Search of the Neural Circuits of Intrinsic Motivation

Frontiers in Neuroscience

Volumn 1, Issue 1, 2007, Pages 225-236

  Kaplan, Frederic ^a   Oudeyer, Pierre Yves ^b  

^a EPFL   (Switzerland)

^b SONY COMPUTER SCIENCE LABORATORY   (France)

Author keywords

cortical microcircuits; curiosity; development; dopamine; exploration; intrinsic motivation; meta learning

Indexed keywords

EID: 85088976837     PISSN: 16624548     EISSN: 1662453X     Source Type: Journal    
DOI: 10.3389/neuro.01.1.1.017.2007     Document Type: Review

References (102)

1. Baldassarre G., (2002).A modular neural-network model of the basal ganglia's role in learning and selection motor behaviors.J. Cogn. Sys. Res.3,5–13.10.1016/S1389-0417(01)00039-0
2. Barto A., (1995).Adaptive critics and the basal ganglia.In Models of information processing in the basal ganglia,Houk J., Davis J., Beiser D., eds.,Cambridge, MA, USA, MIT Press,PP. 215–232.
3. Barto A., Singh S., Chentanez N., (2004).Intrinsically motivated learning of hierarchical collections of skills.In Proceedings of the 3rd International Conference on Development and Learning (ICDL 2004),Salk Institute,San Diego.
4. Bell D., (1973).The experimental reproduction of amphetamine psychosis.Arch. Gen. Psychiatry.29,35–40.
5. Belue M., Bauer K., Ruck D., (1997).Selecting optimal experiments for multiple output multi-layer perceptrons.Neural Comput.9,161–183.10.1162/neco.1997.9.1.161
6. Berlyne D., (1960).Conflict, Arousal and Curiosity(McGraw-Hill).
7. Bernheimer H., Birkmayer W., HornykiewiczJellinger K., Seitelberger F., (1973).Brain dopamine and the syndromes of parkinson and huntington: Clinical, morphological and neurochemical correlations.J. Neurol. Sci.20,415–455.10.1016/0022-510X(73)90175-5
8. Berridge K. C., (2007).The debate over dopamine's role in reward: the case of incentive salience.Psychopharmacology191,391–431.10.1007/s00213-006-0578-x
9. Bruner J., (1962).On Knowing: Essays for the Left Hand(Cambridge, MA, Harvard University Press).
10. Cameron J., Pierce W., (2002).Rewards and Intrinsic Motivation: Resolving the Contoversy(Bergin and Garvey Press).
11. Carboni E., Imperato A., Perezzani L., Di Chiara G., (1989).Amphetamine, cocaine, phencyclidine and nomifensine increases extra-cellullar dopamine concentrations preferentially in the nucleus accumbens of freely moving rats.Neuroscience28,653–661.10.1016/0306-4522(89)90012-2
12. Cohn D., Ghahramani Z., Jordan M., (1996).Active learning with statistical models.J. Artif. Intel. Res.4,129–145.
13. Cordeschi R., (2002).The Discovery of the Artifical. Behavior, Mind and Machines, Before and Beyond Cybernetics(Kluwer academic publishers, Dordrecht).
14. Csikszenthmihalyi M., (1991).Flow-the Psychology of Optimal Experience(Harper Perennial).
15. Daw N., Kakade S., Dayan P., (2002).Opponent interactions between serotonin and dopamine.Neural Netw.15,603–616.10.1016/S0893-6080(02)00052-7
16. Dayan P., Sejnowski T. J., (1996).Exploration bonuses and dual control.Mach. Learn.25,5–22.
17. De Charms R., (1968).Personal Causation: The Internal Affective Determinants of Behavior(New York, Academic Press).
18. Deci E., Ryan R., (1985).Intrinsic Motivation and Self-Determination in Human Behavior.Plenum Press.
19. Dember W. N., Earl R. W., (1957).Analysis of exploratory, manipulatory and curiosity behaviors.Psychol. Rev.64,91–96.10.1037/h0046861
20. Deneve S., Duhamel J.-R., Pouget A., (2007).Optimal sensorimotor integration in recurrent cortical networks: A neural implementation of kalman filters.J. Neurosci.27,5744–5756.10.1523/JNEUROSCI.3985-06.2007
21. Depue R., Iacono W., (1989).Neurobehavioral aspects of affective disorders.Ann. Rev. Psychol.40,457–492.
22. Dommett E., Coizet V., Blatha C., Martindale J., Lefebre V., Walton N., Mayhew J., Overton P., Redgrave P., (2005).How visual stimuli activate dopaminergic neurons at short latency.Science,307,1476–1479.10.1126/science.1107026
23. Douglas R., Martin K., (1998).Neocortex.In The Synaptic Organization of the Brain, Shepherd G.M., ed.,(Oxford University Press)PP. 459–509.
24. Doya K., (1999).What are the computations of cerebellum, basal ganglia, and the cerebral cortex.Neural Netw.12,961–974.10.1016/S0893-6080(99)00046-5
25. Doya K., (2002).Metalearning and neuromodulation.Neural Netw.15,4–5.
26. Doya K., Samejima K., Katagiri K., Kawato M., (2002).Multiple modelbased reinforcement learning.Neural Comput.14,1347–1369.10.1162/089976602753712972
27. Fedorov V., (1972).Theory of Optimal Experiment(New York, NY, Academic Press).
28. Festinger L., (1957).A theory of Cognitive Dissonance(Evanston, Row, Peterson).
29. Fiorillo C. D., (2004).The uncertain nature of dopamine.Mol. Psychiatry,122–123.10.1038/sj.mp.4001445
30. Galantucci B., (2005).An experimental study of the emergence of human communication systems.Cogn. Sci.29,737–767.
31. Gallese V., Lakoff G., (2005).The brainis concepts: role of sensorymotor conceptual knowledge.Cogn. Neuropsychol.21.
32. Grace A. A., (1991).Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia.Neuroscience41,1–24.10.1016/0306-4522(91)90196-U
33. Gray J., (1982).The Neuropsychology of Anxiety: An Enquiry into the Functions of the Septo-Hippocampal System(Oxford, Clarendon Press).
34. Gray J., (1990).Brain systems that mediate both emotion and cognition.Cogn. Emot.4,269–288.
35. Harlow H., (1950).Learning and satiation of response in intrinsically motivated complex puzzle performances by monkeys.J. Comp. Physiolo. Psychology43,289–294.10.1037/h0058114
36. Heath R. G., (1963).Electrical self-stimulation of the brain in man.American J. Psychiatry120,571–577.
37. Hebb D. O., (1955).Drives and the c.n.s (conceptual nervous system).Psycholo. Rev.62,243–254.10.1037/h0041823
38. Hooks M., Kalivas P., (1994).Involvement of dopamine and excitatory amino acid transmission in novelty-induced motor activity.J. Pharmacol Exp. Ther.269,976–988.
39. Horvitz J.-C., (2000).Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events.Neuroscience96,651–656.10.1016/S0306-4522(00)00019-1
40. Horvitz J.-C., (2002).Dopamine gating of glutamatergic sensorimotor and incentive motivational input signals to the striatum.Behav. Brain Res.137,65–74.10.1016/S0166-4328(02)00285-1
41. Houk J., Adams J., Barto A., (1995).A model of how the basal ganglia generate and use neural signals that predict reinforcement.In Models of Information Processing in the Basal Ganglia,Houk J., Davis J., Beiser D., eds.(MIT press)PP. 249–270.
42. Huang X., Weng J., (2002).Novelty and reinforcement learning in the value system of developmental robots.In Proceedings of the 2nd International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems,Prince C., Demiris Y., Marom Y., Kozima H., Balkenius C., eds.Lund University Cognitive Studies 94,PP. 47–55.
43. Hull C. L., (1943).Principles of Behavior: An Introduction to Behavior Theory(New-York: Appleton-Century-Croft).
44. Hunt J. M., (1965).Intrinsic motivation and its role in psychological development.Nebraska Symposium on Motivation,13,189–282.
45. Ikemoto S., Panksepp J., (1999).The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to rewardseeking.Brain Res. Rev.31,6–41.10.1016/S0165-0173(99)00023-5
46. Jaeger H., (2001).The echo state approach to analyzing and training recurrent neural networks. Technical Report, GMD Report 148, GMD - German National Research Institute for Computer Science.
47. Jaeger H., Haas H., (2004).Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication.Science304,78–80.10.1126/science.1091277
48. Joel D., Niv Y., Ruppin E., (2002).Actor-critic models of the basal ganglia: New anatomical and computational perspectives.Neural Netw.15,535–547.10.1016/S0893-6080(02)00047-3
49. Jordan M., Jacobs R., (1994).Hierarchical mixtures of experts and the em algorithm.Neural Comput.6,181–214.
50. Kagan J., (1972).Motives and development.J. Pers. Soc. Psychol.22,51–66.10.1037/h0032356
51. Kakade S., Dayan P., (2002).Dopamine: Generalization and bonuses.Neural Netw.15,549–559.10.1016/S0893-6080(02)00048-5
52. Kaplan F., Oudeyer P.-Y., (2004).Maximizing learning progress: an internal reward system for development.In Embodied Artificial Intelligence,Iida F., Pfeifer R., Steels L., Kuniyoshi Y., eds.LNCS 3139,(Springer-Verlag, London, UK)PP. 259–270.
53. Kaplan F., Oudeyer P.-Y., (2007a).The progress-drive hypothesis: an interpretation of early imitation.In Models and Mechanisms of Imitation and Social Learning: Behavioural, Social and Communication Dimensions,Nehaniv C., Dautenhahn K., eds.(Cambridge University Press)PP. 361–377.
54. Kaplan F., Oudeyer P.-Y., (2007b).Un robot motivé pour apprendre: le role des motivations intrinseques dans le developpement sensorimoteur.Enfance59,46–58.
55. Karmiloff-Smith A., (1992).Beyond Modularity: A Developmental Perspective on Cognitive Science(MIT Press).
56. Kawato M., (1999).Internal models for motor control and trajectory planning.Curr. Opin. Neurobiol.9,718–727.10.1016/S0959-4388(99)00028-8
57. Khamassi M., Lachèze L., Girard B., Berthoz A., Guillot A., (2005).Actor-critic models of reinforcement learning in the basal ganglia.Adapt. Behav.13,131–148.10.1177/105971230501300205
58. Koepp M., Gunn R., Cunningham V., Dagher A., T., J., Brooks D. J., Bench C. J., Grasby P. M., (1998).Evidence for striatal dopamine release during a video game.Nature393,266–267.10.1038/30498
59. Lakoff G., Johnson M., (1998).Philosophy in the Flesh: the Embodied Mind and its Challenge to Western Thought(Basic Books).
60. Maas W., Natschlager T., Markram H., (2002).Real-time computing without stable states: a new framework for neural computation based on perturbations.Neural Comput.14,2531–2560.10.1162/089976602760407955
61. Marshall J., Blank D., Meeden L., (2004).An emergent framework for self-motivation in developmental robotics.In Proceedings of the 3rd International Conference on Development and Learning (ICDL 2004),Salk Institute, San Diego.
62. McClure S., Daw N., Montague P., (2003).A computational substate for incentive salience.Trends Neurosci.26.10.1016/S0166-2236(03)00177-2
63. Montague P., Dayan P., Sejnowski T., (1996).A framework for mesencephalic dopamine systems based on predictive hebbian learning.J. Neurosci.16,1936–1947.
64. Montgomery K., (1954).The role of exploratory drive in learning.J. Comp. Physiol. Psychol.47,60–64.10.1037/h0054833
65. Mountcastle V., (1978).An organizing principle for cerebral function: The unit model and the distributed system.In The Mindful Brain.Edelman G., Mountcastle V., eds.(MIT press).
66. Niv Y., Daw N., Joel D., Dayan P., (2006).Tonic dopamine: opportunity costs and the control of response vigor.Psychopharmacology507–520.
67. Oades R., (1985).The role of noradrenaline in tuning and dopamine in switching between signals in the cns.Neurosci. Biobehav. Rev.9,261–282.10.1016/0149-7634(85)90050-8
68. Oudeyer P.-Y., Kaplan F., (2006).Discovering communication.Connect. Science18,189–206.10.1080/09540090600768567
69. Oudeyer P.-Y., Kaplan F., (2007).What is intrinsic motivation? a typology of computational approaches.Front. Neurorobot.1.
70. Oudeyer P.-Y., Kaplan F., Hafner V., (2007).Intrinsic motivation systems for autonomous mental development.IEEE Trans. Evol. Comput.11,265–286.10.1109/TEVC.2006.890271
71. Panksepp J., (1998).Affective neuroscience: the Foundations of Human and Animal Emotions.(Oxford University Press).
72. Pettit H., Justice J. Jr., (1989).Dopamine in the nucleus accumbens during cocaine self-administration as studied by in vivo microdialysis.Pharmacol. Biochem. Behav.34,899–904.10.1016/0091-3057(89)90291-8
73. Piaget J., (1952).The Origins of Intelligence in Children(New York, NY,Norton).
74. Ploghaus A., Tracey I., Clare S., Gati J., Rawlins J., Matthews P., (2000).Learning about pain: the neural substrate of the prediction error of aversive events.Proc. Natl. Acad. Sci.97,9281–9286.10.1073/pnas.160266497
75. Quaade F., Vaernet K., Larsson S., (1974).Stereotaxic stimulation and electrocoagulation of the lateral hypothalamus in obese humans.Acta. Neurochir.30,111–117.10.1007/BF01405759
76. Redgrave P., Prescott T., Gurney K., (1999).Is the short latency dopamine response too short to signal reward error?Trends Neurosci.22,146–151.10.1016/S0166-2236(98)01373-3
77. Rolls E. T., (1999).The Brain and Emotion(Oxford UP).
78. Rosenblueth A., Wiener N., Bigelow J., (1943).Behavior, purpose and teleology.Philos. Sci.10,18–24.10.1086/286788
79. Roth G., Dicke U., (2005).Evolution of the brain and intelligence.Trends Cogn. Sci.9,250–257.10.1016/j.tics.2005.03.005
80. Ryan R., Deci E., (2000).Intrinsic and extrinsic motivations: Classic definitions and new directions.Contemp. Educ. Psychol.25,54–67.10.1006/ceps.1999.1020
81. Schmidhuber J., (1991).Curious model-building control systems.In Proceeding International Joint Conference on Neural Networks,Singapore. IEEE,Vol. 2,PP. 1458–1463.
82. Schultz W., (1998).Predictive reward signal of dopamine neurons.J. Neurophysiol.80,1–27.
83. Schultz W., (2006).Behavioral theories and the neurophysiology of reward.Annu. Rev. Psychol.57,87–115.10.1146/annurev.psych.56.091103.070229
84. Schultz W., Dayan P., Montague P., (1997).A neural substrate of prediction and reward.Science275,1593–1599.10.1126/science.275.5306.1593
85. Shepherd G. M., (1988).A basic circuit for cortical organization.In Perspectives in Memory Research,Gazzaniga M., ed.(MIT Press)PP. 93–134.
86. Smith A., Li M., Becker S., Kapur S., (2006).Dopamine, prediction error and associative learning: a model-based account.Netw. Comput. Neural Sys.17,61–84.10.1080/09548980500361624
87. Steels L., (2004).The autotelic principle.In Embodied Artificial Intelligence, Fumiya I., Pfeifer R., Steels L., Kunyoshi K., eds.Vol. 3139 of Lecture Notes in AI,(Berlin, Springer Verlag)PP. 231–242.
88. Stellar J., (1985).The Neurobiology of Motivation and Reward(New York, NY, Springer Verlag).
89. Suri R., Schultz W., (2001).Temporal difference model reproduces anticipatory neural activity.Neural Comput.13,841–862.10.1162/089976601300014376
90. Sutton R., (1988).Learning to predict by the methods of temporal differences.Mach. Learn.3,9–44.
91. Sutton R., Barto A., (1998).Reinforcement Learning: An Introduction(Cambridge, MA, MIT Press).
92. Tani J., Nolfi S., (1999).Learning to perceive the world as articulated: An approach for hierarchical learning in sensory-motor systems.Neural Netw.12,1131–1141.10.1016/S0893-6080(99)00060-X
93. Thelen E., Smith L. B., (1994).A Dynamic Systems Approach to the Development of Cognition and Action(Boston, MA, USA, MIT Press).
94. Thorndike E., (1911).Animal Intelligence: Experimental Studies(MacMillan, New York, NY).
95. Thrun S., Pratt L., (1998).Learning to Learn.Kluwer Academic Publishers.
96. Walkins C., Dayan P., (1992).Q-learning.Mach. Learn.8,279–292.
97. Weinberger D. R., (1987).Implications of normal brain development for the pathogenesis of schizophrenia.Arch. Gen. Psychiatry44,660–669.
98. Weiner I., Joel D., (2002).Dopamine in schizophrenia: dysfunctional information processing in basal ganglia-thalamocortical split circuits.In Handbook of Experimental Pharmacology, vol 154/II, Dopamine in the CNS II,Chiara G., ed.(Springer)PP. 417–472.
99. White N. M., (1989).Reward or reinforcement: what's the difference?Neurosci.Biobehav. Rev.13,181–186.10.1016/S0149-7634(89)80028-4
100. White R., (1959).Motivation reconsidered: The concept of competence.Psychol. Rev.66,297–333.10.1037/h0040934
101. Wise R., (1989).The brain and reward.In The Neuropharmacological Basis of Reward,Liebman J., Cooper S., eds.(Clarendon Press)PP. 377–424.
102. Yoshimoto K., McBride W., Lumeng L., Li T.-K., (1991).Alcohol stimulates the release of dopamine and serotonin in the nucleus accumbens.Alcohol.9,17–22.10.1016/0741-8329(92)90004-T