Motivated reinforcement learning: Curious characters for multiuser games

Motivated Reinforcement Learning: Curious Characters for Multiuser Games

Volumn , Issue , 2009, Pages 1-206

  Merrick, Kathryn ^a   Maher, Mary Lou ^b  

^a UNIVERSITY OF NEW SOUTH WALES   (Australia)

^b UNIVERSITY OF SYDNEY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84892360157     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-540-89187-1     Document Type: Book

References (145)

1. R. Bartle, Designing virtual worlds, New Riders, Indianapolis, 2004.
2. P. Baillie-de Byl, Programming believable characters for computer games, Charles River Media, Hingham, Massachusetts, 2004.
3. J. Laird and M. van Lent, Human-level AI's killer application: interactive computer games. AI Magazine, pp. 15-25, Summer 2001.
4. D. Zeltzer, Autonomy, interaction and presence. Presence: Teleoperators and Virtual Environments 1(1):127-132, 1992.
5. Linden, Second Life, www.secondlife.com (Accessed January, 2007).
6. Active Worlds, www.activeworlds.com (Accessed January, 2007).
7. There.com www.there.com (Accessed July, 2008).
8. F. Rex, LambdaMOO: An introduction, http://www.lambdamoo.info (Accessed December, 2006).
9. K. Merrick and M.L. Maher, Motivated reinforcement learning for adaptive characters in open-ended simulation games, ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE 2007), ACM, Salzburg, Austria, pp. 127-134, 2007.
10. M.L. Maher and J.S. Gero, Agent models of 3D virtual worlds, ACADIA 2002: Thresholds, California State Polytechnic University, Pamona, pp. 127-138, 2002.
11. S. Woodcock, Games making interesting use of artificial intelligence techniques. http://www.gameai.com/games.html (Accessed October, 2005).
12. Siege University 303 Skrit, http://garage.gaspowered.com (Accessed March, 2006).
13. D. Johnson and J. Wiles, Computer games with intelligence, The Tenth IEEE International Conference on Fuzzy Systems, pp. 1355-1358, 2001.
14. C. Reynolds, Flocks, herds and schools: A distributed behavioural model. Computer Graphics 21(4):25-34, 1987.
15. N.J. Nilsson, Introduction to machine learning, http://ai.stanford.edu/ people/nilsson/mlbook.html (Accessed January, 2006), 1996.
16. S.J. Russell and P. Norvig, Artificial intelligence: A modern approach, Prentice Hall, Englewood Cliffs, New Jersey, 1995.
17. R.S. Sutton and A.G. Barto, Reinforcement learning: An introduction, The MIT Press Cambridge, Massachusetts, London, England, 2000.
18. T. Graepel, R. Herbrich and J. Gold, Learning to fight, The International Conference on Computer Games: Artificial Intelligence, Design and Education, 2004.
19. D. Goldberg, Genetic algorithms, Addison-Wesley, Reading, Massachusetts, 1989.
20. R.S. Sutton and A.G. Barto, Reinforcement learning: An introduction, The MIT Press Cambridge, Massachusetts, London, England, 2000.
21. D.G. Mook, Motivation: The organization of action, W. W. Norton and Company, New York, 1987.
22. R.G. Geen, W.W. Beatty and R.M. Arkin, Human motivation: physiological, behavioral and social approaches, Allyn and Bacon, Massachusetts, 1984.
23. E.R. Kandel, J.H. Schwarz and T.M. Jessell, Essentials of neural science and behaviour, Appleton and Lang, Norwalk, 1995.
24. C. Alderfer, Existence, relatedness and growth, Free Press, New York, 1972.
25. A. Maslow, Motivation and personality, Harper, New York, 1954.
26. C.L. Hull, Principles of behavior, Appleton-Century-Crofts, New York, 1943.
27. C.L. Hull, A behavior system: An introduction to behavior theory concerning the individual organism, Yale University Press, New Haven, 1952.
28. O. Avila-Garcia and L. Canamero, Comparison of behaviour selection architectures using viability indicators, The EPSRC/BBSRC International Workshop on Biologically Inspired Robotics: The Legacy of W. Grey Walter, HP Labs, Bristol, UK, pp. 8-93, 2002.
29. L. Canamero, Modelling motivations and emotions as a basis for intelligent behaviour. In: W.L. Johnson (Eds.), The First International Symposium on Autonomous Agents, ACM Press, New York, NY, pp. 148-155, 1997.
30. C. Gershenson, Artificial Societies of Intelligent Agents, Engineering, Fundacion Arturo Rosenblueth, Mexico, 2001.
31. R. Aylett, A. Coddington and G. Petley, Agent-based continuous planning, The 19th Workshop of the UK Planning and Scheduling Special Interest Group (PLANSIG 2000), 2000.
32. D. Lukose (Eds.), Multi-Agent Systems: Theories, Languages and Applications, Proceedings of the Fourth Australian Workshop on Distributed Artificial Intelligence, Springer-Verlag, pp. 58-73, 1998.
33. T.J. Norman and D. Long, Goal creation in motivated agents, Intelligent agents: Theories, architectures and languages, Springer-Verlag, pp. 277-290, 1995.
34. M. Schmill and P. Cohen, A motivational system that drives the development of activity AAMAS, ACM, Bologna, Italy, pp. 358-359, 2002.
35. D. McFarland, Animal behaviour, Longman, England, 1995.
36. D. Bindra, A unified account of classical conditioning and operant training. In: A. Black and W. Prokasy (Eds.), Classical conditioning-current theory and research, Appleton-Century-Crofts, New York, USA, pp. 453-481, 1974.
37. W. Wundt, Principles of physiological psychology, MacMillan, New York, 1910.
38. D.E. Berlyne, Novelty, complexity and hedonic value. Perception and Psychophysics 8:279-286, 1970.
39. D.E. Berlyne, Aesthetics and psychobiology, Prentice-Hall, Englewood Cliffs, New Jersey, 1971.
40. D.E. Berlyne, Exploration and curiosity. Science 153:25-33, 1966.
41. S. Marsland, U. Nehmzow and J. Shapiro, A real-time novelty detector for a mobile robot, EUREL European Advanced Robotics Systems Masterclass and Conference, 2000.
42. J.A. Easterbrook, The effect of emotion on cue utilization and the organisation of behavior. Psychological Review 66:183-201, 1959.
43. D. Sperber and D. Wilson, Relevance: communication and cognition, Blackwell Publishing, Oxford, 1995.
44. L. Fonar and P. Maes, Paying attention to what's important: using focus of attention to improve unsupervised learning, The Third International Conference on the Simulation of Adaptive Behavior (SAB94), Brighton, England, 1994.
45. J.C. Stanley, Computer simulation of a model of habituation. Nature 26:146-148, 1976.
46. T. Kohonen, Self-organization and associative memory, Springer, Berlin, 1993.
47. R. Saunders and J.S. Gero, Curious agents and situated design evaluations. In: J.S. Gero and F.M.T. Brazier (Eds.), Agents in Design, Key Centre of Design Computing and Cognition, University of Sydney, pp. 133-149, 2002.
48. R. Saunders and J.S. Gero, Situated design simulations using curious agents. AIEDAM 18(2):153-161, 2004.
49. D. Helbing and P. Molnar, Social force model for pedestrian dynamics. Physical Review E 51:4282-4286, 1995.
50. R. Saunders and J.S. Gero, Designing for interest and novelty: motivating design agents, CAAD Futures 2001, Kluwer, Dordrecht, pp. 725-738, 2001.
51. J. Schmidhuber, A possibility for implementing curiosity and boredom in modelbuilding neural controllers. In: J.A. Meyer and S.W. Wilson (Eds.), The International Conference on Simulation of Adaptive Behavior: From Animals to Animats, MIT Press/Bradford Books, pp. 222-227, 1991.
52. D. Lenat,: An artificial intelligence approach to discovery in mathematics, Computer Science, Stanford University, 1976.
53. J. Schmidhuber, What's interesting, Technical Report, Lugano, Switzerland, 1997.
54. P.-Y. Oudeyer and F. Kaplan, Intelligent adaptive curiosity: A source of selfdevelopment, Fourth International Workshop on Epigenetic Robotics, Lund University, pp. 127-130, 2004.
55. P.-Y. Oudeyer, F. Kaplan and V. Hafner, Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11(2):265-286, 2007.
56. F. Kaplan and P.-Y. Oudeyer, Motivational principles for visual know-how development. In: C.G. Prince, L. Berthouze, H. Kozima, D. Bullock, G. Stojanov and C. Balkenius (Eds.), Proceedings of the 3rd international workshop on Epigenetic Robotics: Modelling cognitive development in robotic systems, Lund University Cognitive Studies, pp. 73-80, 2003.
57. E.C. Tolman, Purposive behavior in animals and men, Century, New York, 1932.
58. B.F. Skinner, Behavior of organisms, Appleton-Century-Crofts, New York, 1938.
59. J.W. Atkinson and N.T. Feather, A theory of achievement motivation, John Wiley, New York, 1966.
60. J.W. Atkinson and J.O. Raynor, Motivation and achievement, V. H. Winston, Washington D. C, 1974.
61. J.O. Raynor, Future orientation and motivation of immediate activity: An elaboration of the theory of achievement motivation. Psychological Review 76:606-610, 1969.
62. F. Heider, The psychology of interpersonal relations, Wiley, New York, 1958.
63. R.W. White, Motivation reconsidered: The concept of competence. Psychological Review 66:297-333, 1959.
64. E. Deci and R. Ryan, Intrinsic motivation and self-determination in human behavior, Plenum Press, New York, 1985.
65. J.M. Hunt, Implications of sequential order and hierarchy in early psychological development. Exceptional Infant 3, 1975.
66. S. Singh, A.G. Barto and N. Chentanez, Intrinsically motivated reinforcement learning, Advances in Neural Information Processing Systems (NIPS) 17:1281-1288, 2005.
67. D. Precup, R. Sutton and S. Singh, Theoretical results on reinforcement learning with temporally abstract options, The 10th European Conference on Machine Learning, Chemnitz, Germany, Springer Verlag, pp. 382-393, 1998.
68. M. Csikszentmihalyi, Creativity: Flow and the psychology of discovery and invention, Harper Collins Publisher, New York, 1996.
69. R. Saunders and J.S. Gero, The digital clockwork muse: A computational model of aesthetic evolution, The AISB'01 Symposium on Artificial Intelligence and Creativity in Arts and Science SSAISB, 2001.
70. C. Darwin, The origin of the species, John Murray, London, 1859.
71. R.V. Gusarev, M.S. Burtsev and V.G. Red'ko, Alife model of evolutionary emergence of purposeful adaptive behavior, European Conference on Artificial Life, 2001.
72. R. Stagner, Homeostasis, discrepancy, dissonance: A theory of motives and motivation. Motivation and Emotion 1:103-138, 1977.
73. R.A. Brooks, How to build complete creatures rather than isolated cognitive simulators. In: K. VanLehn (Ed.), Architectures for Intelligence, Lawrence Erlbaum Associates, New Jersey, pp. 225-239, 1991.
74. N.J. Nilsson, Introduction to machine learning, http://ai.stanford.edu/ people/nilsson/mlbook.html (Accessed January, 2006), 1996.
75. R.S. Sutton and A.G. Barto, Reinforcement learning: An introduction, The MIT Press, 2000.
76. T. Graepel, R. Herbrich and J. Gold, Learning to fight, The International Conference on Computer Games: Artificial Intelligence, Design and Education, 2004.
77. A.G. Barto and S. Mahadevan, Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems: Theory and Applications 13(4):41-77, 2003.
78. C. Watkins and P. Dayan, Q-learning. Machine Learning 8(3):279-292, 1992.
79. D. Precup, R. Sutton and S. Singh, Theoretical results on reinforcement learning with temporally abstract options, The 10th European Conference on Machine Learning, Chemnitz, Germany, Springer Verlag, pp. 382-393, 1998.
80. S.J. Bradtke and M.O. Duff, Reinforcement learning methods for continuous-time Markov decision problems. In: G. Tesauro, D.S. Touretzky, and T. Leen, (Eds.), Advances in Neural Information Processing Systems: Proceedings of the 1994 Conference, MIT Press, Cambridge, MA, pp. 393-400, 1995.
81. R. Parr, Hierarchical control and learning for Markov Decision Processes, University of California, Berkeley, 1998.
82. R. Sutton, D. Precup and S. Singh, Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112:181-211, 1998.
83. A. McGovern, Autonomous discovery of temporal abstractions from interaction with an environment, PhD Thesis, DePARTment of Computer Science, University of Massachusetts, 2002.
84. B. Hengst, Discovering hierarchy in reinforcement learning with HEXQ, The 19th International Conference on Machine Learning, University of New South Wales, Sydney, Australia, pp. 243-250, 2002.
85. O. Simsek and A.G. Barto, Using relative novelty to identify useful temporal abstractions in reinforcement learning, The 21st International Conference on Machine Learning, Banff, Canada, 2004.
86. X. Huang and J. Weng, Inherent value systems for autonomous mental development, International Journal of Humanoid Robotics, 4(2):407-433, 2007.
87. J. Weng and W. Hwang, An incremental learning algorithm with automatically derived discriminating features, Fourth Asian Conference on Computer Vision, Taipei, Taiwan, pp. 426-431, 2000.
88. J. Schmidhuber, A possibility for implementing curiosity and boredom in modelbuilding neural controllers. In J.A. Meyer, and S.W. Wilson (Eds.), The International Conference on Simulation of Adaptive Behavior: From Animals to Animats, MIT Press/Bradford Books, pp. 222-227, 1991.
89. J. Schmidhuber, What's interesting, Lugano, Switzerland, 1997.
90. O. Simsek and A.G. Barto, An intrinsic reward mechanism for efficient exploration, The 23rd International Conference on Machine Learning, University of Pittsburgh, Pennsylvania, USA, 2006.
91. S. Singh, A.G. Barto and N. Chentanez, Intrinsically motivated reinforcement learning, Advances in Neural Information Processing Systems (NIPS) 17:1281-1288, 2005.
92. A.G. Barto, S. Singh and N. Chentanez, Intrinsically motivated learning of hierarchical collections of skills, International Conference on Developmental Learning, 2004.
93. R. Saunders and J.S. Gero, Designing for interest and novelty: motivating design agents, CAAD Futures 2001, Kluwer, Dordrecht, pp. 725-738, 2001.
94. S. Marsland, U. Nehmzow and J. Shapiro, A real-time novelty detector for a mobile robot, EUREL European Advanced Robotics Systems Masterclass and Conference, 2000.
95. T. Kohonen, Self-organization and associative memory, Springer, Berlin, 1993.
96. J.C. Stanley, Computer simulation of a model of habituation. Nature 261:146-148.
97. F. Kaplan and P.-Y. Oudeyer, Motivational principles for visual know-how development. In: C.G. Prince, L. Berthouze, H. Kozima, D. Bullock, G. Stojanov and C. Balkenius (Eds.), Proceedings of the 3rd International Workshop on Epigenetic Robotics: Modelling Cognitive Development in Robotic Systems, Lund University Cognitive Studies, pp. 73-80, 2003.
98. P.-Y. Oudeyer, F. Kaplan and V. Hafner, Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11(2):265-286, 2007.
99. G.N. Yannakakis and J. Hallam, Towards capturing and enhancing entertainment in computer games, The Fourth Hellenic Conference on Artificial Intelligence, Springer-Verlag, pp. 432-442, 2006.
100. P. Rani, N. Sarkar and C. Liu, Maintaining optimal challenge in computer games through real-time physiological feedback, HCI International, Las Vegas, USA, 2005.
101. M. Csikszentmihalyi, Flow: The psychology of optimal experience, Harper Perennial, 1990.
102. J. Chen, Flow in games (and everything else). Communications of the ACM 50(4):31-34, 2007.
103. J. Chen, Flow in Games. http://www.jenovachen.com/flowingames/about.htm (Accessed July, 2008).
104. M. Duggan, The official guide to 3D game studio, Thomson Course Technology, 2007.
105. E. Byrne, Game level design, Cengage Delmar Learning, 2004.
106. A. Doull, The death of the level designer: procedural content generation in games. http://roguelikedeveloper.blogspot.com/2008/01/death-of-level- designerprocedural.html (Accessed July, 2008).
107. L. Kaelbling, M. Littman and A. Moore, Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4:237-285, 1996.
108. D. Berry and B. Fristedt, Bandit problems: sequential allocation of experiments, Chapman and Hall, London, 1985.
109. R.S. Sutton and A.G. Barto, Reinforcement learning: An introduction, The MIT Press Cambridge, Massachusetts, London, England, 2000.
110. B. Hengst, Discovering hierarchy in reinforcement learning with HEXQ, The 19th International Conference on Machine Learning, University of New South Wales, Sydney, Australia, pp. 243-250, 2002.
111. S. Singh, A.G. Barto and N. Chentanez, Intrinsically motivated reinforcement learning, Advances in Neural Information Processing Systems (NIPS), 17:1281-1288, 2005.
112. J. Schmidhuber, Exploring the predictable. In A. Ghosh, S. Tsutsui (Eds.), Advances in Evolutionary Computing, pp. 579-612, Springer, 2002.
113. F. Kaplan and P.-Y. Oudeyer, Motivational principles for visual know-how development. In: C.G. Prince, L. Berthouze, H. Kozima, D. Bullock, G. Stojanov and C. Balkenius (Eds.), Proceedings of the 3rd international workshop on Epigenetic Robotics: Modelling cognitive development in robotic systems, Lund University Cognitive Studies, pp. 73-80, 2003.
114. X. Huang and J. Weng, Inherent value systems for autonomous mental development, International Journal of Humanoid Robotics, 4(2):407-433, 2007.
115. R. Saunders and J.S. Gero, Designing for interest and novelty: motivating design agents, CAAD Futures 2001, Kluwer, Dordrecht, pp. 725-738, 2001.
116. M. Wooldridge and N. Jennings, Intelligent agents: Theory and practice. The Knowledge Engineering Review 10:115-152, 1995.
117. A. Merceron, Languages and logic, Pearson Education Australia, 2001.
118. S. Singh, A.G. Barto and N. Chentanez, Intrinsically motivated reinforcement learning, Advances in Neural Information Processing Systems (NIPS), 17:1281-1288, 2005.
119. R. Saunders, Curious design agents and artificial creativity, Faculty of Architecture, University of Sydney, Sydney, 2001.
120. S. Marsland, U. Nehmzow and J. Shapiro, A real-time novelty detector for a mobile robot, EUREL European Advanced Robotics Systems Masterclass and Conference, 2000.
121. J.C. Stanley, Computer simulation of a model of habituation. Nature 261:146-148, 1976.
122. R.W. White, Motivation reconsidered: The concept of competence. Psychological Review 66:297-333, 1959.
123. E. Deci and R. Ryan, Intrinsic motivation and self-determination in human behavior, Plenum Press, New York, 1985.
124. G.A. Carpenter and S. Grossberg, The ART of adaptive pattern recognition by a self-organizing neural network. IEEE Computer 21:77-88, 1988.
125. J. Laird and M. vanLent, Interactive computer games: human-level AI's killerapplication, National Conference on Artificial Intelligence (AAAI), pp. 1171-1178, 2000.
126. D. Precup, R. Sutton and S. Singh, Theoretical results on reinforcement learning with temporally abstract options, The 10th European Conference on Machine Learning, Chemnitz, Germany, Springer Verlag, pp. 382-393, 1998.
127. R. Saunders and J.S. Gero, Curious agents and situated design evaluations. In: J.S. Gero and F.M.T. Brazier (Eds.), Agents in Design, Key Centre of Design Computing and Cognition, University of Sydney, pp. 133-149, 2002.
128. M.L. Maher and J.S. Gero, Agent models of 3D virtual worlds, ACADIA 2002: Thresholds, California State Polytechnic University, Pamona, pp. 127-138, 2002.
129. R. Sutton, D. Precup and S. Singh, Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112:181-211, 1998.
130. Linden, Second Life, www.secondlife.com (Accessed January, 2007).
131. K. Merrick and M.L. Maher, Motivated reinforcement learning for non-player characters in persistent computer game worlds, International Conference on Advances in Computer Entertainment Technology, 2006, CA, USA (electronic proceedings).
132. K. Merrick, Modelling motivation for adaptive non-player characters in dynamic computer game worlds. ACM Computers in Entertainment 5(4), 2007.
133. Linden, Second Life, www.secondlife.com (Accessed January, 2007).
134. There.com, www.there.com (Accessed July, 2008).
135. K. Merrick and M.L. Maher, Motivated reinforcement learning for adaptive characters in open-ended simulation games, ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, (ACE 2007), ACM, Salzburg, Austria, pp. 127-134, 2007.
136. O. Macindoe, M.L. Maher and K. Merrick, Agent Based Intrinsically Motivated Intelligent Environments, Handbook on Mobile and Ubiquitous Computing: Innovations and Perspectives, American Scientific Publishers, 2008 (to appear).
137. J. Chen, Flow in games. http://www.jenovachen.com/flowingames/about.htm, (Accessed July, 2008).
138. A. Doull, The death of the level designer: procedural content generation in games. http://roguelikedeveloper.blogspot.com/2008/01/death-of-level- designerprocedural.html (Accessed July, 2008).
139. N. Gu and M.L. Maher, A grammar for the dynamic design of virtual architecture using rational agents. International Journal of Architectural Computing 4(1):489-501, 2004.
140. R. Saunders and J.S. Gero, Curious agents and situated design evaluations. In: J.S. Gero and F.M.T. Brazier (Eds.), Agents in Design, Key Centre of Design Computing and Cognition, University of Sydney, pp. 133-149, 2002.
141. A. Maslow, Motivation and personality, Harper, New York, 1954.
142. R.A. Brooks, How to build complete creatures rather than isolated cognitive simulators. In: K. VanLehn (Ed.), Architectures for Intelligence, Lawrence Erlbaum Associates, NJ, pp. 225-239, 1991.
143. R.A. Brooks, M. Coen, D. Dang, J. DeBonet, J. Kramer, T. Lozano-Perez, J. Mellor, P. Pook, C. Stauffer, L. Stein, M. Torrance and M. Wessler, The intelligent room project, The Second International Cognitive Technology Conference (CT97), Aizu, pp. 271-279, 1997.
144. K. Merrick, M.L. Maher and R. Saunders, Achieving adaptable behaviour in intelligent rooms using curious supervised learning agents, CAADRIA 2008, Beyond Computer Aided Design, Chiang Mai, Thailand, pp. 185-192, 2008.
145. M.L. Maher, K. Merrick and O. Macindoe, Intrinsically motivated intelligent sensed environments, EGICE 2006, Springer-Verlag, pp. 455-475, 2006.