Intrinsic motivation and reinforcement learning

Intrinsically Motivated Learning in Natural and Artificial Systems

Volumn 9783642323751, Issue , 2013, Pages 17-47

  Barto, Andrew G ^a,b  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b INSTITUTE OF COGNITIVE SCIENCES AND TECHNOLOGIES   (Italy)

Author keywords

[No Author keywords available]

Indexed keywords

MOTIVATION; ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS; REINFORCEMENT LEARNING;

EXTRINSIC MOTIVATION; HIGH GRADES; INPUT CHANNELS; INTRINSIC MOTIVATION; LEARN+; LEARNING FRAMEWORKS; MACHINE LEARNING SYSTEMS; MACHINE-LEARNING; REINFORCEMENT LEARNING AGENT; REINFORCEMENT LEARNINGS;

REINFORCEMENT LEARNING; EDUCATION;

EID: 84929046579     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-642-32375-1_2     Document Type: Chapter

References (108)

1. Ackley, D.H., Littman, M.: Interactions between learning and evolution. In: Langton, C., Taylor, C., Farmer, C., Rasmussen, S. (eds.) Artificial Life II (Proceedings Volume X in the Santa Fe Institute Studies in the Sciences of Complexity, pp. 487-509. Addison-Wesley, Reading (1991)
2. Andry, P., Gaussier, P., Nadel, J., Hirsbrunner, B.: Learning invariant sensorimotor behaviors: A developmental approach to imitation mechanisms. Adap. Behav. 12, 117-140 (2004)
3. Arkes, H.R., Garske, J.P.: Psychological Theories of Motivation. Brooks/Cole, Monterey (1982)
4. Baranes, A., Oudeyer, P.-Y.: Intrinsically motivated goal exploration for active motor learning in robots: A case study. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2010), Taipei, Taiwan 2010
5. Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discr. Event Dynam. Syst. Theory Appl. 13, 341-379 (2003)
6. Barto, A.G., Singh, S., Chentanez, N.: Intrinsically motivated learning of hierarchical collections of skills. In: Proceedings of the International Conference on Developmental Learning (ICDL), La Jolla, CA 2004
7. Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike elements that can solve difficult learn-ingcontrol problems. 13, 835-846 (1983). IEEE Trans. Sys. Man, Cybern.
8. Reprinted in J.A. Anderson and E. Rosenfeld (eds.), Neurocomputing: Foundations of Research, pp. 535-549, MIT, Cambridge (1988)
9. Beck, R.C.: Motivation. Theories and Principles, 2nd edn. Prentice-Hall, Englewood Cliffs (1983)
10. Berlyne, D.E.: A theory of human curiosity. Br. J. Psychol. 45, 180-191 (1954)
11. Berlyne, D.E.: Conflict, Arousal., Curiosity. McGraw-Hill, New York (1960)
12. Berlyne, D.E.: Curiosity and exploration. Science 143, 25-33 (1966)
13. Berlyne, D.E.: Aesthetics and Psychobiology. Appleton-Century-Crofts, New York (1971)
14. Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena Scientific, Belmont (1996)
15. Bindra, D.: How adaptive behavior is produced: A perceptual-motivational alternative to response reinforcement. Behav. Brain Sci. 1, 41-91 (1978)
16. Breazeal, C., Brooks, A., Gray, J., Hoffman, G., Lieberman, J., Lee, H., Lockerd, A., Mulanda, D.: Tutelage and collaboration for humanoid robots. Int. J. Human. Robot. 1 (2004)
17. Bush, V.: Science the endless frontier: Areport to the president. Technical report (1945)
18. Busoniu, L., Babuska, R., Schutter, B.D.: A comprehensive survey of multi-agent reinforcement learning. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 38(2), 156-172 (2008)
19. Cannon, W.B.: The Wisdom of the Body. W.W. Norton, New York (1932)
20. Clark, W.A., Farley, B.G.: Generalization of pattern recognition in a self-organizing system. In: AFIPS' 55 (Western) Proceedings of the March 1-3, 1955, Western Joint Computer Conference, Los Angeles, CA, pp. 86-91, ACM, New York (1955)
21. Cofer, C.N., Appley, M.H.: Motivation: Theory and Research. Wiley, New York (1964)
22. Damoulas, T., Cos-Aguilera, I., Hayes, G.M., Taylor, T.: Valency for adaptive homeostatic agents: Relating evolution and learning. In: Capcarrere, M.S., Freitas, A.A., Bentley, P.J., Johnson, C.G., Timmis, J. (eds.) Advances in Artificial Life: 8th European Conference, ECAL 2005. Canterbury, UK LNAI Vol. 3630, pp. 936-945. Springer, Berlin (2005)
23. Daw, N.D., Shohamy, D.: The cognitive neuroscience of motivation and learning. Soc. Cogn. 26(5), 593-620 (2008)
24. Dayan, P.: Motivated reinforcement learning. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference, pp. 11-18. MIT, Cambridge (2001)
25. Deci, E.L., Ryan, R.M.: Intrinsic Motivation and Self-Determination in Human Behavior. Plenum, New York (1985)
26. Dember, W.N., Earl, R.W.: Analysis of exploratory, manipulatory, and curiosity behaviors. Psychol. Rev. 64, 91-96 (1957)
27. Dember, W.N., Earl, R.W., Paradise, N.: Response by rats to differential stimulus complexity. J. Comp. Physiol. Psychol. 50, 514-518 (1957)
28. Dickinson, A., Balleine, B.: The role of leaning in the operation of motivational systems. In: Gallistel, R. (ed.) Handbook of Experimental Psychology, 3rd edn. Learning, Motivation, and Emotion, pp. 497-533. Wiley, New York (2002)
29. Elfwing, S., Uchibe, E., Doya, K., Christensen, H.I.: Co-evolution of shaping rewards and metaparameters in reinforcement learning. Adap. Behav. 16, 400-412 (2008)
30. Epstein, A.: Instinct and motivation as explanations of complex behavior. In: Pfaff, D.W. (ed.) The Physiological Mechanisms of Motivation. Springer, New York (1982)
31. Friston, K.J., Daunizeau, J., Kilner, J., Kiebel, S.J.: Action and behavior: A free-energy formulation. Biol. Cybern. (2010). Pubished online February 11, 2020
32. Groos, K.: The Play of Man. D. Appleton, New York (1901)
33. Harlow, H.F.: Learning and satiation of response in intrinsically motivated complex puzzle performance by monkeys. J. Comp. Physiol. Psychol. 43, 289-294 (1950)
34. Harlow, H.F., Harlow, M.K., Meyer, D.R.: Learning motivated by a manipulation drive. J. Exp. Psychol. 40, 228-234 (1950)
35. Hart, S., Grupen, R.: Intrinsically motivated affordance discovery and modeling. In: Baldassarre, G., Mirolli, M. (eds.) Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, Berlin (2012, this volume)
36. Hebb, D.O.: The Organization of Behavior. Wiley, New York (1949)
37. Hendrick, I.: Instinct and ego during infancy. Psychoanal. Quart. 11, 33-58 (1942)
38. Hesse, F., Der, R., Herrmann, M., Michael, J.: Modulated exploratory dynamics can shape self-organized behavior. Adv. Complex Syst. 12(2), 273-292 (2009)
39. Hull, C.L.: Principles of Behavior. D. Appleton-Century, New York (1943)
40. Hull, C.L.: Essentials of Behavior. Yale University Press, New Haven (1951)
41. Hull, C.L.: A Behavior System: An Introduction to Behavior Theory Concerning the Individual Organism. Yale University Press, New Haven (1952)
42. Kimble, G.A.: Hilgard and Marquis' Conditioning and Learning. Appleton-Century-Crofts, Inc., New York (1961)
43. Klein, S.B.: Motivation. Biosocial Approaches. McGraw-Hill, New York (1982)
44. Klopf, A.H.: Brain function and adaptive systems - A heterostatic theory. Technical report AFCRL-72-0164, Air Force Cambridge Research Laboratories, Bedford. A summary appears in Proceedings of the International Conference on Systems, Man, and Cybernetics, 1974, IEEE Systems, Man, and Cybernetics Society, Dallas (1972)
45. Klopf, A.H.: The Hedonistic Neuron: A Theory of Memory, Learning, and Intelligence. Hemisphere, Washington (1982)
46. Lenat, D.B.: AM: An artificial intelligence approach to discovery in mathematics. Ph.D. Thesis, Stanford University (1976)
47. Linden, D.J.: The Compass of Pleasure: How Our Brains Make Fatty Foods, Orgasm, Exercise, Marijuana, Generosity, Vodka, Learning, and Gambling Feel So Good. Viking, New York (2011)
48. Littman, M.L., Ackley, D.H.: Adaptation in constant utility nonstationary environments. In: Proceedings of the Fourth International Conference on Genetic Algorithms, San Diego, CA pp. 136-142 (1991)
49. Lungarella, M., Metta, G., Pfeiffer, R., Sandini, G.: Developmental robotics: A survey. Connect. Sci. 15, 151-190 (2003)
50. Mackintosh, N.J.: Conditioning and Associative Learning. Oxford University Press, New York (1983)
51. McFarland, D., Bösser, T.: Intelligent Behavior in Animals and Robots. MIT, Cambridge (1993)
52. Mendel, J.M., Fu, K.S. (eds.): Adaptive, Learning, and Pattern Recognition Systems: Theory and Applications. Academic, New York (1970)
53. Mendel, J.M., McLaren, R.W.: Reinforcement learning control and pattern recognition systems. In: Mendel, J.M., Fu, K.S. (eds.) Adaptive, Learning and Pattern Recognition Systems: Theory and Applications, pp. 287-318. Academic, New York (1970)
54. Michie, D., Chambers, R.A.: BOXES: An experiment in adaptive control. In: Dale, E., Michie, D. (eds.) Machine Intelligence 2, pp. 137-152. Oliver and Boyd, Edinburgh (1968)
55. Minsky, M.L.: Theory of neural-analog reinforcement systems and its application to the brain-model problem. Ph.D. Thesis, Princeton University (1954)
56. Minsky, M.L.: Steps toward artificial intelligence. Proc. Inst. Radio Eng. 49, 8-30 (1961).
57. Reprinted in E.A. Feigenbaum and J. Feldman (eds.) Computers and Thought, pp. 406-450. McGraw-Hill, New York (1963)
58. Mollenauer, S.O.: Shifts in deprivations level: Different effects depending on the amount of preshift training. Learn. Motiv. 2, 58-66 (1971)
59. Narendra, K., Thathachar, M.A.L.: Learning Automata: An Introduction. Prentice Hall, Englewood Cliffs (1989)
60. Olds, J., Milner, P.: Positive reinforcement produced by electrical stimulation of septal areas and other regions of rat brain. J. Comp. Physiol. Psychol. 47, 419-427 (1954)
61. Oudeyer, P.-Y., Kaplan, F.: What is intrinsic motivation? A typology of computational approaches. Front. Neurorobot. 1:6, doi: 10.3389/neuro.12.006.2007 (2007)
62. Oudeyer, P.-Y., Kaplan, F., Hafner, V.: Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput. 11, 265-286 (2007)
63. Petri, H.L.: Motivation: Theory and Research. Wadsworth Publishing Company, Belmont (1981)
64. Piaget, J.: The Origins of Intelligence in Children. Norton, New York (1952)
65. Picard, R.W.: Affective Computing. MIT, Cambridge (1997)
66. Prince, C.G., Demiris, Y., Marom, Y., Kozima, H., Balkenius, C. (eds.): Proceedings of the Second International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund University Cognitive Studies, Vol. 94. Lund University, Lund (2001)
67. Rescorla, R.A., Wagner, A.R.: A theory of Pavlovian conditioning: Variationsin the effectiveness of reinforcement and nonreinforcement. In: Black, A.H., Prokasy, W.F. (eds.) Classical Conditioning, Vol. II, pp. 64-99. Appleton-Century-Crofts, New York (1972)
68. Rosenblatt, F.: Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Spartan Books, Washington (1962)
69. Rumelhart, D., Hintont, G., Williams, R.: Learning representations by back-propagating errors. Nature 323 (6088), 533-536 (1986)
70. Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemp. Educ. Psychol. 25, 54-67 (2000)
71. Samuelson, L.: Introduction to the evolution of preferences. J. Econ. Theory 97, 225-230 (2001)
72. Samuelson, L., Swinkels, J.: Information, evolution, and utility. Theor. Econ. 1, 119-142 (2006)
73. Savage, T.: Artificial motives: A review of motivation in artificial creatures. Connect. Sci. 12, 211-277 (2000)
74. Schembri, M., Mirolli, M., Baldassarre, G.: Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot. In: Proceedings of the 6th International Conference on Development and Learning (ICDL2007), Imperial College, London 2007
75. Schmidhuber, J.: Adaptive confidence and adaptive curiosity. Technical report FKI-149-91, Institut für Informatik, Technische Universität München (1991a)
76. Schmidhuber, J.: A possibility for implementing curiosity and boredom in model-building neural controllers. In: From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, pp. 222-227. MIT, Cambridge (1991b)
77. Schmidhuber, J.: What's interesting? Technical report TR-35-97. IDSIA, Lugano (1997)
78. Schmidhuber, J.: Artificial curiosity based on discovering novel algorithmic predictability through coevolution. In: Proceedings of the Congress on Evolutionary Computation, Vol. 3, pp. 1612-1618. IEEE (1999)
79. Schmidhuber, J.: Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes. In: Pezzulo, G., Butz, M.V., Sigaud, O., Baldassarre, G. (eds.) Anticipatory Behavior in Adaptive Learning Systems. From Psychological Theories to Artificial Cognitive Systems, pp. 48-76. Springer, Berlin (2009)
80. Schultz, W.: Predictive reward signal of dopamine neurons. J. Neurophysiol. 80(1), 1-27 (1998)
81. Schultz, W.: Reward. Scholarpedia 2(3), 1652 (2007a)
82. Schultz, W.: Reward signals. Scholarpedia 2(6), 2184 (2007b)
83. Scott, P.D., Markovitch, S.: Learning novel domains through curiosity and conjecture. In: Sridharan, N.S. (ed.) Proceedings of the 11th International Joint Conference on Artificial Intelligence, Detroit, MI pp. 669-674. Morgan Kaufmann, San Francisco (1989)
84. Settles, B.: Active learning literature survey. Technical Report 1648, Computer Sciences, University of Wisconsin-Madison, Madison (2009)
85. Singh, S., Barto, A.G., Chentanez, N.: Intrinsically motivated reinforcement learning. In: Advances in Neural Information Processing Systems 17: Proceedings of the 2004 Conference. MIT, Cambridge (2005)
86. Singh, S., Lewis, R.L., Barto, A.G.: Where do rewards come from? In: Taatgen, N., van Rijn, H. (eds.) Proceedings of the 31st Annual Conference of the Cognitive Science Society, Amsterdam pp. 2601-2606. Cognitive Science Society (2009)
87. Singh, S., Lewis, R.L., Barto, A.G., Sorg, J.: Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Trans. Auton. Mental Dev. 2(2), 70-82 (2010). Special issue on Active Learning and Intrinsically Motivated Exploration in Robots: Advances and Challenges
88. Snel, M., Hayes, G.M.: Evolution of valence systems in an unstable environment. In: Proceedings of the 10th International Conference on Simulation of Adaptive Behavior: From Animals to Animats, Osaka, M. Asada, J.C. Hallam, J.-A. Meyer (Eds.) pp. 12-21 (2008)
89. Sorg, J., Singh, S., Lewis, R.L.: Internal rewards mitigate agent boundedness. In: Fürnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, Omnipress pp. 1007-1014 (2010)
90. Sutton, R.S.: Reinforcement learning architectures for animats. In: From Animals to Animats: Proceedings of the First International Conference on Simulation of Adaptive Behavior, J.-A. Meyer, S.W. Wilson (Eds.) pp. 288-296. MIT, Cambridge (1991)
91. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT, Cambridge (1998)
92. Sutton, R.S., Precup, D., Singh, S.: Between mdps and semi-mdps: A framework for temporal abstraction inreinforcement learning. Artif. Intell. 112, 181-211 (1999)
93. Tesauro, G.J.: TD - gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6(2), 215-219 (1994)
94. Thomaz, A.L., Breazeal, C.: Transparency and socially guided machine learning. In: Proceedings of the 5th International Conference on Developmental Learning (ICDL) Bloomington, IN (2006)
95. Thomaz, A.L., Hoffman, G., Breazeal, C.: Experiments in socially guided machine learning: Understanding how humans teach. In: Proceedings of the 1st Annual conference on HumanRobot Interaction (HRI) Salt Lake City, UT (2006)
96. Thorndike, E.L.: Animal Intelligence. Hafner, Darien (1911)
97. Toates, F.M. (1911): Motivational Systems. Cambridge University Press, Cambridge (1911)
98. Tolman, E.C.: Purposive Behavior in Animals and Men. Naiburg, New York (1932)
99. Trappl, R., Petta, P., Payr, S. (eds.): Emotions in Humans and Artifacts. MIT, Cambridge (1997)
100. Uchibe, E., Doya, K.: Finding intrinsic rewards by embodied evolution and constrained reinforcement learning. Neural Netw. 21(10), 1447-1455 (2008)
101. Waltz, M.D., Fu, K.S.: A heuristic approach to reinforcement learning control systems. IEEE Transactions on Automatic Control 10, 390-398 (1965)
102. Weng, J., McClelland, J., Pentland, A., Sporns, O., Stockman, I., Sur, M., Thelen, E.: Autonomous mental development by robots and animals. Science 291, 599-600 (2001)
103. Werbos, P.J.: Building and understanding adaptive systems: A statistical/numerical approach to factory automation and brain research. IEEE Trans. Sys. Man Cybern. 17, 7-20 (1987)
104. White, R.W.: Motivation reconsidered: The concept of competence. Psychol. Rev. 66, 297-333 (1959)
105. Widrow, B., Gupta, N.K., Maitra, S.: Punish/reward: Learning with a critic in adaptive thresh-oldsystems. IEEE Trans. Sys. Man Cybern. 3, 455-465 (1973)
106. Widrow, B., Hoff, M.E.: Adaptive switching circuits. In: 1960 WESCON Convention Record Part IV, pp. 96-104. Institute of Radio Engineers, New York (1960).
107. Reprinted in J.A. Anderson and E. Rosenfeld, Neurocomputing: Foundations of Research, pp. 126-134. MIT, Cambridge (1988)
108. Young, P.T.: Hedonic organization and regulation of behavior. Psychol. Rev. 73, 59-86 (1966)