Formal theory of creativity, fun, and intrinsic motivation (1990-2010)

IEEE Transactions on Autonomous Mental Development

Volumn 2, Issue 3, 2010, Pages 230-247

  Schmidhuber, Jrgen ^a  

^a UNIVERSITY OF LUGANO   (Switzerland)

Author keywords

Active learning; aesthetics theory; art; attention; developmental psychology; formal theory of creativity; fun; humor; limited computational resources; music; novel patterns; novelty; science; surprise; typology of intrinsic motivation

Indexed keywords

ACTIVE LEARNING; AESTHETICS THEORY; ART; ATTENTION; COMPUTATIONAL RESOURCES; DEVELOPMENTAL PSYCHOLOGY; FORMAL THEORY; FUN; HUMOR; INTRINSIC MOTIVATION; MUSIC; NOVEL PATTERNS; NOVELTY; SCIENCE; SURPRISE;

MOTIVATION; OPTIMIZATION;

DATA COMPRESSION;

EID: 77956578648     PISSN: 19430604     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAMD.2010.2056368     Document Type: Article

References (120)

1. B. Bakker and J. Schmidhuber, F. Groen, Ed. et al., "Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization," in Proc. 8th Conf. Intell. Autonom. Syst., Amsterdam, The Netherlands, 2004, pp. 438-445.
2. M. F. Balcan, A. Beygelzimer, and J. Langford, "Agnostic active learning," J. Comput. Syst. Sci., vol.75, no.1, pp. 78-89, 2009.
3. A. G. Barto, S. Singh, and N. Chentanez, "Intrinsically motivated learning of hierarchical collections of skills," in Proc. Int. Conf. Develop. Learn., Cambridge, MA, 2004.
4. M. Bense, Einführung in die Informationstheoretische Ästhetik. Grundlegung und Anwendung in der Texttheorie (Introduction to Information-Theoretical Aesthetics. Foundation and Application to Text Theory). Berlin, Germany: Rowohlt Taschenbuch Verlag, 1969.
5. D. E. Berlyne, "Novelty and curiosity as determinants of exploratory behavior," Brit. J. Psychol., vol.41, pp. 68-80, 1950.
6. D. E. Berlyne, Conflict, Arousal, and Curiosity. New York: Mc-Graw-Hill, 1960.
7. G. D. Birkhoff, Aesthetic Measure. Cambridge, MA: Harvard Univ. Press, 1933.
8. C. M. Bishop, Neural Networks for Pattern Recognition. London, U.K.: Oxford Univ. Press, 1995.
9. D. Blank and L. Meeden, "Introduction to the special issue on developmental robotics," Connect. Sci., vol.18, no.2, 2006.
10. R. I. Brafman and M. Tennenholtz, "R-MAX-A general polynomial time algorithm for near-optimal reinforcement learning," J. Mach. Learn. Res., vol.3, pp. 213-231, 2002.
11. R. Cilibrasi and P. Vitanyi, "Clustering by compression," IEEE Trans. Inf. Theory, vol.51, no.4, pp. 1523-1545, Apr. 2005.
12. D. A. Cohn, "Neural network exploration using optimal experiment design," in Advances in Neural Information Processing Systems, J. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, vol.6, pp. 679-686.
13. P. Dayan and G. Hinton, "Feudal reinforcement learning," in Advances in Neural Information Processing Systems, D. S. Lippman, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1993, vol.5, pp. 271-278.
14. K. Doya, K. Samejima, K. Katagiri, and M. Kawato, "Multiple model-based reinforcement learning," Neural Comput., vol.14, no.6, pp. 1347-1369, 2002.
15. V. V. Fedorov, Theory of Optimal Experiments. New York: Academic, 1972.
16. H. G. Frank, Kybernetische Analysen Subjektiver Sachverhalte. Quickborn, Germany: Verlag Schnelle, 1964.
17. H. G. Frank and H. W. Franke, Ästhetische Information. Estetika informacio. Eine Einführung in die Kybernetische Ästhetik. Berlin, Germany: Kopäd Verlag, 2002.
18. H. W. Franke, Kybernetische Ästhetik. Phänomen Kunst, 3rd ed. Munich, Germany: Ernst Reinhardt Verlag, 1979.
19. K. J. Friston, J. Daunizeau, J. Kilner, and S. J. Kiebel, "Action and behavior: A free-energy formulation," Biol. Cybern., vol.102, no.3, pp. 227-260, 2010.
20. K. Gold and B. Scassellati, "Learning acceptable windows of contingency," Connect. Sci., vol.18, no.2, 2006.
21. F. J. Gomez, "Sustaining diversity using behavioral information distance," in Proc. Conf. Genet. Evol. Comput., Montreal, QC, Canada, 2009, pp. 113-120.
22. F. J. Gomez, J. Schmidhuber, and R. Miikkulainen, "Efficient nonlinear control through neuroevolution," J. Mach. Learn. Res., vol.9, pp. 937-965, 2008.
23. F. J. Gomez, J. Togelius, and J. Schmidhuber, "Measuring and optimizing behavioral complexity," in Proc. Int. Conf. Artif. Neural Netw., Lamissol, Cyprus, 2009, pp. 765-774.
24. A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, and J. Schmidhuber, "A novel connectionist system for improved unconstrained handwriting recognition," IEEE Trans. Pattern Anal. Mach. Intell., vol.31, no.5, May 2009.
25. H. F. Harlow, M. K. Harlow, and D. R. Meyer, "Novelty and curiosity as determinants of exploratory behavior," J. Exp. Psychol., vol.41, pp. 68-80, 1950.
26. S. Hart, "The Development of Hierarchical Knowledge in Robot Systems," Ph.D. dissertation, Dept. Comput. Sci., Univ. Massachusetts, Amherst, MA, 2009.
27. S. Hart, S. Sen, and R. Grupen, "Intrinsically motivated hierarchical manipulation," in Proc. IEEE Conf. Robot. Autom., Pasadena, CA, 2008.
28. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol.9, no.8, pp. 1735-1780, 1997.
29. J. H. Holland, Adaptation in Natural and Artificial Systems. Ann Arbor, MI: Univ. Michigan Press, 1975.
30. J. H. Holland, "Properties of the bucket brigade," in Proc. Int. Conf. Genet. Algorithms, Hillsdale, NJ, 1985.
31. D. A. Huffman, "A method for construction of minimum-redundancy codes," Proc. IRE, vol.40, pp. 1098-1101, 1952.
32. M. Hutter, Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Berlin, Germany: Springer-Verlag, 2004.
33. J.Hwang, J. Choi, S. Oh, and R. J. Marks, II, "Query-based learning applied to partially trained multilayer perceptrons," IEEE Trans. Neural Netw., vol.2, pp. 131-136, Feb. 1991.
34. L. Itti and P. F. Baldi, "Bayesian surprise attracts human attention," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2005, vol.19, pp. 547-554.
35. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. AI Res., vol.4, pp. 237-285, 1996.
36. A. S. Klyubin, D. Polani, and C. L. Nehaniv, "Empowerment: A universal agent-centric measure of control," in Proc. Int. Conf. E-Commerce, Müchen, Germany, 2005.
37. A. N. Kolmogorov, "Three approaches to the quantitative definition of information," Problems Inform. Trans., vol.1, pp. 1-11, 1965.
38. B. Kuipers, P. Beeson, J. Modayil, and J. Provost, "Bootstrap learning of foundational representations," Connect. Sci., vol.18, no.2, 2006.
39. S. Kullback, Statistics and Information Theory. New York: Wiley, 1959.
40. J. Lehman and K. O. Stanley, "Exploiting open-endedness to solve problems through the search for novelty," in Proc. 11th Int. Conf. Artif. Life, Cambridge, MA, 2008.
41. D. B. Lenat, "Theory formation by heuristic search," Mach. Learn., vol.21, 1983.
42. D. B. Lenat and J. S. Brown, "Why AM an EURISKO appear to work," Artif. Intell., vol.23, no.3, pp. 269-294, 1984.
43. L. A. Levin, "Universal sequential search problems," Problems Inform. Trans., vol.9, no.3, pp. 265-266, 1973.
44. L. Li, M. L. Littman, and T. J.Walsh, "Knows what it knows: A framework for self-aware learning," in Proc. 25th Int. Conf. Mach. Learn., Helsinki, Finland, 2008.
45. M. Li and P. M. B. Vitányi, An Introduction to Kolmogorov Complexity and its Applications, 2nd ed. Berlin, Germany: Springer-Verlag, 1997.
46. M. L. Littman and D. H. Ackley, R. K. Belew and L. Booker, Eds., "Adaptation in constant utility non-stationary environments," in Proc. 4th Int. Conf. Genet. Algorithms, San Mateo, CA, 1991, pp. 136-142.
47. D. J. C. MacKay, "Information-based objective functions for active data selection," Neural Comput., vol.4, no.2, pp. 550-604, 1992.
48. A. Moles, Information Theory and Esthetic Perception. Chicago, IL: Univ. Illinois Press, 1968.
49. A. Moore and C. Atkeson, "The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces," Mach. Learn., vol.21, 1995.
50. F. Nake, Ästhetik als Informationsverarbeitung. Berlin, Germany: Springer-Verlag, 1974.
51. L. Olsson, C. L. Nehaniv, and D. Polani, "From unknown sensors and actuators to actions grounded in sensorimotor perceptions," Connect. Sci., vol.18, no.2, 2006.
52. P.-Y. Oudeyer and F. Kaplan, "What is intrinsic motivation? A typology of computational approaches," Frontiers Neurorobot., vol.1, 2006.
53. P.-Y. Oudeyer, F. Kaplan, and V. F. Hafner, "Intrinsic motivation systems for autonomous mental development," IEEE Trans. Evol. Comput., vol.11, pp. 265-286, Nov. 2007.
54. J. Piaget, The Child's Construction of Reality. London, U.K.: Routledge Kegan Paul, 1955.
55. M. Plutowski, G. Cottrell, and H. White, "Learning Mackey-Glass from 25 examples, plus or minus 2," in Advances in Neural Information Processing Systems, J. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, vol.6, pp. 1135-1142.
56. J. Poland and M. Hutter, "Strong asymptotic assertions for discrete MDL in regression and classification," in Proc. Annu. Mach. Learn. Conf. Belgium Netherlands, Enschede, The Netherlands, 2005.
57. J. Provost, B. J. Kuipers, and R. Miikkulainen, "Developing navigation behavior through self-organizing distinctive state abstraction," Connect. Sci., vol.18, no.2, 2006.
58. V. Raskin, Semantic Mechanisms of Humor. Dordrecht, Germany: Springer-Verlag, 1985.
59. I. Rechenberg, "Evolutionsstrategie-Optimierung Technischer Systeme Nach Prinzipien der Biologischen Evolution," Ph.D. dissertation, Technische Universität Berlin, Berlin, Germany, 1971.
60. M. B. Ring, L. Birnbaum and G. Collins, Eds., "Incremental development of complex behaviors through automatic construction of sensory-motor hierarchies," in Proc. 8th Int. Workshop Mach. Learn. , Evanston, IL, 1991, pp. 343-347.
61. M. B. Ring, "Continual Learning in Reinforcement Environments," Ph.D. dissertation, Univ. Texas, Austin, TX, 1994.
62. J. Rissanen, "Modeling by shortest data description," Automatica, vol.14, pp. 465-471, 1978.
63. M. Schembri, M. Mirolli, and G. Baldassarre, Y. Demiris, B. Scassellati, and D. Mareschal, Eds., "Evolving internal reinforcers for an intrinsically motivated reinforcement-learning robot," in Proc. 6th IEEE Int. Conf. Develop. Learn., London, U.K., 2007, pp. 282-287.
64. M. Schlesinger, "Decomposing infants' object representations: A dualroute processing account," Connect. Sci., vol.18, no.2, 2006.
65. J. Schmidhuber, "A local learning algorithm for dynamic feedforward and recurrent networks," Connect. Sci., vol.1, no.4, pp. 403-412, 1989.
66. J. Schmidhuber, "The neural bucket brigade," in Connectionism in Perspective, R. Pfeifer, Z. Schreter, Z. Fogelman, and L. Steels, Eds. Amsterdam, The Netherlands: Elsevier, 1989, pp. 439-446.
67. J. Schmidhuber, "Dynamische neuronale Netze und das fundamentale raumzeitliche Lernproblem," Ph.D. dissertation, Technische Universität München, München, Germany, 1990.
68. J. Schmidhuber, "An on-line algorithm for dynamic reinforcement learning and planning in reactive environments," in Proc. IEEE/INNS Int. Joint Conf. Neural Netw., San Diego, CA, 1990, vol.2, pp. 253-258.
69. J. Schmidhuber, Adaptive Curiosity and Adaptive Confidence Technische Universität München, Institut für Informatik, 1991, Tech. Rep. FKI-149-191
70. J. Schmidhuber, "Curious model-building control systems," in Proc. Int. Joint Conf. Neural Netw., Singapore, 1991, vol.2, pp. 1458-1463.
71. J. Schmidhuber, J. A. Meyer and S. W. Wilson, Eds., "A possibility for implementing curiosity and boredom in model-building neural controllers," in Proc. Int. Conf. Simulation Adapt. Behav.: From Animals to Animats, 1991, pp. 222-227.
72. J. Schmidhuber, "Reinforcement learning in Markovian and non-Markovian environments," in Advances in Neural Information Processing Systems 3, D. S. Lippman, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1991, vol.NIPS 3, pp. 500-506.
73. J. Schmidhuber, "Learning complex, extended sequences using the principle of history compression," Neural Comput., vol.4, no.2, pp. 234-242, 1992.
74. J. Schmidhuber, "A computer scientist's view of life, the universe, and everything," in Foundations of Computer Science: Potential-Theory- Cognition, C. Freksa, M. Jantzen, and R. Valk, Eds. Berlin, Germany: Springer-Verlag, 1997, vol.1337, Lecture Notes in Computer Science, pp. 201-208.
75. J. Schmidhuber, Femmes Fractales, 1997.
76. J. Schmidhuber, "Low-complexity art," Leonardo, J. Int. Soc. Arts, Sci. Technol., vol.30, no.2, pp. 97-103, 1997.
77. J. Schmidhuber, What's Interesting? IDSIA, 1997, Technical Report IDSIA-35-97.
78. J. Schmidhuber, Algorithmic Theories of Everything IDSIA, Manno, Lugano, Switzerland, 2000, Tech. Rep. IDSIA-20-100, quant-ph/0011122.
79. J. Schmidhuber, "Exploring the predictable," in Advances in Evolutionary Computing, A. Ghosh and S. Tsuitsui, Eds. Berlin, Germany: Springer-Verlag, 2002, pp. 579-612.
80. J. Schmidhuber, "Hierarchies of generalized Kolmogorov complexities and nonenumerable universal measures computable in the limit," Int. J. Foundations Comput. Sci., vol.13, no.4, pp. 587-612, 2002.
81. J. Schmidhuber, J. Kivinen and R. H. Sloan, Eds., "The Speed Prior: A new simplicity measure yielding near-optimal computable predictions," in Proc. 15th Annu. Conf. Comput. Learn. Theory, Sydney, Australia, 2002, pp. 216-228.
82. J. Schmidhuber, "Optimal ordered problem solver," Mach. Learn., vol.54, pp. 211-254, 2004.
83. J. Schmidhuber, "Completely self-referential optimal reinforcement learners," in Artificial Neural Networks: Biological Inspirations-ICANN 2005, W. Duch, J. Kacprzyk, E. Oja, and S. Zadrozny, Eds. Berlin, Germany: Springer-Verlag, 2005, vol.3697, Lecture Notes in Computer Science, Plenary Talk, pp. 223-233.
84. J. Schmidhuber, "Gödel machines: Towards a technical justification of consciousness," in Adaptive Agents and Multi-Agent Systems III, D. Kudenko, D. Kazakov, and E. Alonso, Eds. Berlin, Germany: Springer-Verlag, 2005, vol.3394, Lecture Notes in Computer Science, pp. 1-23.
85. J. Schmidhuber, "Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts," Connect. Sci., vol.18, no.2, pp. 173-187, 2006.
86. J. Schmidhuber, "Gödel machines: Fully self-referential optimal universal self-improvers," in Artificial General Intelligence, B. Goertzel and C. Pennachin, Eds. Berlin, Germany: Springer-Verlag, 2006, pp. 199-226.
87. J. Schmidhuber, "The new AI: General & sound & relevant for physics," in Artificial General Intelligence, B. Goertzel and C. Pennachin, Eds. Berlin, Germany: Springer-Verlag, 2006, pp. 175-198.
88. J. Schmidhuber, "Randomness in physics," Nature, vol.439, no.3, p. 392, 2006.
89. J. Schmidhuber, "2006: Celebrating 75 years of AI-History and outlook: The next 25 years," in 50 Years of Artificial Intelligence,M. Lungarella, F. Iida, J. Bongard, and R. Pfeifer, Eds. Berlin, Germany: Springer-Verlag, 2007, vol.4850, Lecture Notes in Artificial Intelligence, pp. 29-41.
90. J. Schmidhuber, "Alle berechenbaren Universen (All computable universes)," Transl.: German Spektrum derWissenschaft Spezial, no.3, pp. 75-79, 2007.
91. J. Schmidhuber, "New millennium AI and the convergence of history," in Challenges to Computational Intelligence, W. Duch and J. Mziuk, Eds. Berlin, Germany: Springer-Verlag, 2007, vol.63, Studies in Computational Intelligence, pp. 15-36.
92. J. Schmidhuber, "Simple algorithmic principles of discovery, subjective beauty, selective attention, curiosity & creativity," in Proc. 10th Int. Conf. Discovery Sci., Sendai, Japan, 2007, vol.4755, Lecture Notes in Artificial Intelligence, pp. 26-38.
93. J. Schmidhuber, "Driven by compression progress," in Knowledge-Based Intelligent Information and Engineering Systems KES-2008, I. Lovrek, R. J. Howlett, and L. C. Jain, Eds. Berlin, Germany: Springer-Verlag, 2008, vol.5177, Lecture Notes in Computer Science, p. 11.
94. J. Schmidhuber, "Art & science as by-products of the search for novel patterns, or data compressible in unknown yet learnable ways," in Multiple Ways to Design Research. Research Cases that Reshape the Design Discipline, M. Botta, Ed. Berlin, Germany: Springer-Verlag, 2009, pp. 98-112.
95. J. Schmidhuber, Compression Progress: The Algorithmic Principle Behind Curiosity and Creativity (With Applications of the Theory of Humor), 40 min Video of Invited Talk at Singularity Summit 2009 New York City, 2009 [Online]. Available: http://www.vimeo.com/7441291
96. J. Schmidhuber, "Driven by compression progress: A simple principle explains essential aspects of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes," in Anticipatory Behavior in Adaptive Learning Systems. From Psychological Theories to Artificial Cognitive Systems, G. Pezzulo, M. V. Butz, O. Sigaud, and G. Baldassarre, Eds. Berlin, Germany: Springer-Verlag, 2009, vol.5499, Lecture Notes in Computer Science, pp. 48-76.
97. J. Schmidhuber, "Simple algorithmic theory of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes," SICE J. Soc. Instrument Contr. Eng., vol.48, no.1, pp. 21-32, 2009.
98. J. Schmidhuber, "Ultimate cognition à la Gödel," Cogn. Comput., vol.1, no.2, pp. 177-193, 2009.
99. J. Schmidhuber, "Artificial scientists & artists based on the formal theory of creativity," in Artificial General Intelligence, M. Hutter, Ed. et al. Berlin, Germany: Springer-Verlag, 2010.
100. J. Schmidhuber, A. Graves, F. J. Gomez, and S. Hochreiter, How to Learn Programs with Artificial Recurrent Neural Networks. Cambridge, U.K.: Cambridge Univ. Press, 2010, to be published.
101. J. Schmidhuber and S. Heil, "Sequential neural text compression," IEEE Trans. Neural Netw., vol.7, pp. 142-146, Jul. 1996.
102. J. Schmidhuber and R. Wahnsiedler, J. A. Meyer, H. L. Roitblat, and S. W.Wilson, Eds., "Planning simple trajectories using neural subgoal generators," in Proc. 2nd Int. Conf. Simulation Adapt. Behav., Honolulu, HI, Dec. 7-11, 1992, pp. 196-202.
103. J. Schmidhuber, J. Zhao, and M.Wiering, "Shifting inductive bias with success-story algorithm, adaptive Levin search, and incremental selfimprovement," Mach. Learn., vol.28, pp. 105-130, 1997.
104. H. P. Schwefel, "Numerische Optimierung von Computer-Modellen," Ph.D. dissertation, Technische Universität Berlin, Berlin, Germany, 1974/1975, Reprinted by Birkhäuser.
105. H. S. Seung, M. Opper, and H. Sompolinsky, "Query by committee," in Proc. 5th Annu.Workshop Comput. Learn. Theory, New York, 1992, pp. 287-294.
106. C. E. Shannon, "A mathematical theory of communication (Parts I and II)," Bell Syst. Techn. J., vol.XXVII, pp. 379-423, 1948.
107. S. Singh, A. G. Barto, and N. Chentanez, "Intrinsically motivated reinforcement learning," in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 2005, vol.17, NIPS.
108. S. Singh, R. L. Lewis, and A. G. Barto, N. Taatgen and H. van Rijn, Eds., "Where do rewards come from?," in Proc. 31st Annu. Conf. Cogn. Sci. Soc., Austin, TX, 2009.
109. R. J. Solomonoff, "A formal theory of inductive inference. Part I," Inform. Contr., vol.7, pp. 1-22, 1964.
110. R. J. Solomonoff, "Complexity-based induction systems," IEEE Trans. Inf. Theory, vol.IT-24, pp. 422-432, 1978.
111. J. Storck, S. Hochreiter, and J. Schmidhuber, "Reinforcement driven information acquisition in non-deterministic environments," in Proc. Int. Conf. Artif. Neural Netw., Paris, France, 1995, vol.2, pp. 159-164.
112. A. Strehl, J. Langford, and S. Kakade, Learning from Logged Implicit Exploration Data 2010, Technical Report arXiv:1003.0120.
113. D. Stronger and P. Stone, "Towards autonomous sensor and actuator model induction on a mobile robot," Connect. Sci., vol.18, no.2, 2006.
114. R. Sutton and A. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
115. C. S. Wallace and D. M. Boulton, "An information theoretic measure for classification," Comput. J., vol.11, no.2, pp. 185-194, 1968.
116. C. S.Wallace and P. R. Freeman, "Estimation and inference by compact coding," J. Roy. Statist. Soc., Series B, vol.49, no.3, pp. 240-265, 1987.
117. C. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3/4, pp. 279-292, 1992.
118. M. Wiering and J. Schmidhuber, "Q-learning," Adapt. Behav., vol.6, no.2, pp. 219-246, 1998.
119. M. A.Wiering and J. Schmidhuber, "Fast online Q(λ)," Mach. Learn., vol.33, no.1, pp. 105-116, 1998.
120. S. W. Wilson, "ZCS: A zeroth level classifier system," Evol. Comput., vol.2, pp. 1-18, 1994.