Real-time neuroevolution in the NERO video game

IEEE Transactions on Evolutionary Computation

Volumn 9, Issue 6, 2005, Pages 653-668

  Stanley, Kenneth O ^a   Bryant, Bobby D ^a   Miikkulainen, Risto ^a  

^a University of Texas at Austin   (United States)

Author keywords

Interactive; Neural networks; Neuroevolution; Neuroevolution of augmenting topologies (NEAT); Neuroevolving robotic operatives (NERO); Online; Real time; Video games

Indexed keywords

EVOLUTIONARY ALGORITHMS; INTERACTIVE COMPUTER GRAPHICS; LEARNING SYSTEMS; NEURAL NETWORKS; ONLINE SYSTEMS; REAL TIME SYSTEMS;

NEUROEVOLUTION; NEUROEVOLUTION OF AUGMENTING TOPOLOGIES; NEUROEVOLVING ROBOTIC OPERATIVES (NERO); VIDEO GAMES;

GAME THEORY;

EID: 29244482280     PISSN: 1089778X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TEVC.2005.856210     Document Type: Article

References (77)

1. P. Thurrott. (2002, Jan.) Top stories of 2001, #9: Expanding video-game market brings Microsoft home for the holidays. Windows NET Mag. Netw. [Online]. Available: http://www.winnetmag.com/Article/A.rticleID/23862/ 23862.html
2. J. E. Laird and M. van Lent, "Human-level AI's killer application: Interactive computer games," in Proc. 17th Nat. Conf. Artif. Intell., and 12th Ann. Conf. Innov. Appl. Artif. Intell., 2000, pp. 1171-1178.
3. D. B. Fogel, T. J. Hays, and D. R. Johnson, "A platform for evolving characters in competitive games," in Proc. Congr. Evol. Comput., 2004, pp. 1420-1426.
4. K. O. Stanley and R. Miikkulainen, "Evolving neural networks through augmenting topologies," Evol. Comput., vol. 10, no. 2, pp. 99-127, 2002.
5. K. O. Stanley and R. Miikkulainen, "Competitive coevolution through evolutionary complexification," J. Artif. Intell. Res., vol. 21, pp. 63-100, 2004.
6. A. L. Samuel, "Some studies in machine learning using the game of checkers," IBM J., vol. 3, pp. 210-229, 1959.
7. R. S. Sutton, "Learning to predict by the methods of temporal differences," Mach. Learn., vol. 3, pp. 9-44, 1988.
8. D. Michie, "Trial and error," Penguin Sci. Survey, vol. 2, pp. 129-145, 1961.
9. M. Gardner, "How to build a game-learning machine and then teach it to play and to win," Sci. Amer., vol. 206, no. 3, pp. 138-144, 1962.
10. G. Tesauro and T. J. Sejnowski, "A 'neural' network that learns to play backgammon," in Neural Information Processing Systems, D. Z. Anderson, Ed. New York: Amer. Ins. Physics, 1987.
11. N. Richards, D. Moriarty, P. McQuesten, and R. Miikkulainen, "Evolving neural networks to play Go," in Proc. 7th Int. Conf. Genetic Algorithms, T. Bäck, Ed., East Lansing, MI, 1997, pp. 768-775.
12. K. O. Stanley and R. Miikkulainen, "Evolving a roving eye for Go," in Proc. Genetic Evol. Computa. Conf, 2004, pp. 1226-1238.
13. T. Yoshioka, S. Ishii, and M. Ito, "Strategy acquisition for the game Othello based on reinforcement learning," in Proc. 5th Int. Conf. Neural Inf. Process., S. Usui and T. Omori, Eds. Tokyo, 1998, pp. 841-844.
14. J. Fürnkranz, "Machine leaming in games: A survey," Machines that Learn to Play Games, pp. 11-59, 2001.
15. D. B. Fogel, Blondie24: Playing at the Edge of AI. San Mateo, CA: Morgan Kaufmann, 2001.
16. D. B. Fogel, T. J. Hays, S. L. Hahn, and J. Quon, "A self-learning evolutionary chess program," Proc. IEEE, vol. 92, no. 12, pp. 1947-1954, 2004.
17. B. Geisler, "An empirical study of machine learning algorithms applied to modeling player behavior in a 'first person shooter' video game," M.S. thesis, Dept. Comput. Sci., Univ. Wisconsin-Madison, Madison, WI, 2002.
18. N. Cole, S. Louis, and C. Miles, "Using a genetic algorithm to tune first-person shooter bots," in Proc. Congr. Evol. Comput., vol. 1, 2004, pp. 139-145.
19. J.-H. Hong and S.-B. Cho, "Evolution of emergent behaviors for shooting game characters in robocode," in Proc. Congr. Evol. Comput., vol. 1, 2004, pp. 634-638.
20. M. Gallagher and A. Ryan, "Learning to play Pac-Man: An evolutionary, rule-based approach," in Proc. Congr. Evol, Comput., vol. 4, 2003, pp. 2462-2469.
21. T. Revello and R. McCartney, "Generating war game strategies using a genetic algorithm," in Proc. Congr. Evol. Comput., vol. 2, 2002, pp. 1086-1091.
22. B. D. Bryant and R. Miikkulainen, "Neuroevolution for adaptive teams," in Proc. Congr. Evol. Comput., vol. 3, 2003, pp. 2194-2201.
23. G. Yannakakis, J. Levine, and J. Hallam, "An evolutionary approach for interactive computer games," in Proc. Congr, Evol. Comput., vol. 1, 2004, pp. 986-993.
24. I. Parmee and C. Bonham, "Toward the support of innovative conceptual design through interactive designer/evolutionary computing strategies," Artif. Intell. Eng. Design, Analy. Manuf. J., vol. 14, pp. 3-16, 1999.
25. M. van Lent and J. E. Laird, "Learning procedural knowledge through observation," in Proc. Int. Conf. Knowl. Capture, 2001, pp. 179-186.
26. D. E. Rumelhart and J. L. McClelland, Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge, MA: MIT Press, 1986, vol. 1. PDP Research Group, Foundations.
27. P. E. Utgoff, "Incremental induction of decision trees," Mach. Learn., vol. 4, no. 2, pp. 161-186, 1989.
28. C. J. C. H. Watkins and P. Dayan, "Q-learning," Mach. Learn., vol. 8, no. 3, pp. 279-292, 1992.
29. J. C. Santamaria, R. S. Sutton, and A. Ram, "Experiments with reinforcement learning in problems with continuous state and action spaces," Adapt. Behav, vol. 6, no. 2, pp. 163-218, 1998.
30. L. P. Kaelbling, M. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell., vol. 4, pp. 237-285, May 1996.
31. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press, 1998.
32. F. J. Gomez, "Robust non-linear control through neuroevolution," Ph.D. dissertation, Dept. Comput. Sci., Univ. Texas at Austin, Austin, TX, 2003. Tech. Rep. AI-TR-03-303.
33. F. J. Gomez and R. Miikkulainen, "Active guidance for a finless rocket through neuroevolution," in Proc. Genetic Evol. Comput. Conf., 2003, pp. 2084-2095.
34. R. Aharonov-Barki, T. Beker, and E. Ruppin, "Emergence of memory-Driven command neurons in evolved artificial agents," Neural Comput., vol. 13, no. 3, pp. 691-716, 2001.
35. D. Floriano and F. Mondada, "Automatic creation of an autonomous agent: Genetic evolution of a neural-network driven robot," in Proc. 3rd Int. Conf. Simul. Adapt. Behav. (From Animals to Animats 3), 1994, pp. 421-430.
36. F. Gomez and R. Miikkulainen, "2-D pole-balancing with recurrent evolutionary networks," Proc. Int. Conf. Artif. Neural Netw., pp. 425-430, 1998.
37. F. Gomez and R. Miikkulainen, "Solving non-Markovian control tasks with neuroevolution," in Proc. l6th Int. Joint Conf. Artif. Intell., 1999, pp. 1356-1361.
38. F. Gruau, D. Whitley, and L. Pyeatt, "A comparison between cellular encoding and direct encoding for genetic neural networks," in Proc. 1st Ann. Conf. Genetic Program., J. R. Koza, D. E. Goldberg, D. B. Fogel, and R. L. Riolo, Eds., 1996, pp. 81-89.
39. I. Harvey, "The artificial evolution of adaptive behavior," Ph.D. dissertation, School of Cognitive and Comput. Sci., Univ. Sussex, Sussex, U.K., 1993.
40. D. E. Moriarty and R. Miikkulainen, "Evolving obstacle avoidance behavior in a robot arm," in Proc. 4th Int. Conf. Simul. Adapt. Behav. (From Animals to Animats 4), P. Maes, M. J. Mataric, J.-A. Meyer, J. Pollack, and S. W. Wilson, Eds., 1996, pp. 468-475.
41. S. Nolfi, J. L. Elman, and D. Parisi, "Learning and evolution in neural networks," Adapt. Behav, vol. 2, pp. 5-28, 1994.
42. M. A. Potter, K. A. De Jong, and J. J. Grefenstette, "A coevolutionary, approach to learning sequential decision rules," in Proc. 6th Int. Conf. Genetic Algorithms, L. J. Eshelman, Ed., 1995, pp. 366-372.
43. D. Whitley, S. Dominic, R. Das, and C. W. Anderson, "Genetic reinforcement learning for neurocontrol problems," Mach. Learn., vol. 13, pp. 259-284, 1993.
44. G. Cybenko, "Approximation by superpositions of a sigmoidal function," Math. Control, Signals, and Systems, vol. 2, no. 4, pp. 303-314, 1989.
45. H. T. Siegelmann and E. D. Sontag, "Analog computation via neural networks," Theor. Comput. Sci., vol. 131, no. 2, pp. 331-360, 1994.
46. X. Yao, "Evolving artificial neural networks," Proc. IEEE, vol. 87, no. 9, pp. 1423-1447, 1999.
47. D. E. Moriarty and R. Miikkulainen, "Efficient reinforcement learning through symbiotic evolution," Mach. Learn., vol. 22, pp. 11-32, 1996.
48. N. Saravanan and D. B. Fogel, "Evolving neural control systems," IEEE Expert, pp. 23-27, Jun. 1995.
49. A. Wieland, "Evolving neural network controllers for unstable systems," in Proc. Int. Joint Conf. Neural Netw., Seattle, WA, 1991, pp. 667-673.
50. J. C. Bongard and R. Pfeifer, "Repeated structure and dissociation of genotypic and phenotypic complexity in artificial ontogeny," in Proc. Genetic Evol. Comput. Conf., L. Spector, E. D. Goodman, A. Wu, W. B. Langdon, H.-M. Voigt, M. Gen, S. Sen, M. Dorigo, S. Pezeshk, M. H. Garzon, and E. Burke, Eds., 2001, pp. 829-836.
51. G. S. Hornby and J. B. Pollack, "Creating high-level components with a generative representation for body-brain evolution," Artif. Life, vol. 8, no. 3, pp. 223-246, 2002.
52. M. Mandischer, "Representation and evolution of neural networks, Artif. Neural Nets and Genetic Algorithms, pp. 643-649, 1993.
53. "School of Comput. Sci.," Univ. Birmingham, Birmingham, Birmingham, U.K., Tech. Rep. CSRP-97-7, 1997.
54. B.-T. Zhang and H. Muhlenbein, "Evolving optimal neural networks using genetic algorithms with Occam's razor," Complex Syst., vol. 7, pp. 199-220, 1993.
55. H. Braun and J. Weisbrod, "Evolving feed-forward neural networks," in Proc. ANNGA93, Int. Conf. Artif. Neural Netw. Genetic Algorithms, 1993, pp. 25-32.
56. D. W. Opitz and J. W. Shavlik, "Connectionist theory refinement: Genetically searching the space of network topologies," J. Artif. Intell. Res., vol. 6, pp. 177-209, 1997.
57. X. Yao and Y. Liu, "Toward designing artificial neural networks by evolution," Appl. Math. Comput., vol. 91, no. 1, pp. 83-90, 1996.
58. P. J. Angeline, G. M. Saunders, and J. B. Pollack, "An evolutionary algorithm that constructs recurrent neural networks," IEEE Trans. Neural Netw., vol. 5, pp. 54-65, 1993.
59. R. Krishnan and V. B. Ciesielski, "2DELTA-GANN: A new approach to training neural networks using genetic algorithms," in Proc. Australian Conf. Neural Netw., 1994, pp. 194-197.
60. V. Maniezzo, "Genetic evolution of the topology and weight distribution of neural networks," IEEE Trans. Neural Netw., vol. 5, no. 1, pp. 39-53, Jan. 1994.
61. C.-H. Lee and J.-H. Kim, "Evolutionary ordered neural network with a linked-list encoding scheme," in Proc. IEEE Int. Conf. Evol. Comput., 1996, pp. 665-669.
62. D. Dasgupta and D. McGregor, "Designing application-specific neural networks using the structured genetic algorithm," in Proc. Int. Conf. Combinations Genetic Algorithms Neural Netw., 1992, pp. 87-96.
63. D. Moriarty and R. Miikkulainen, "Evolving complex Othello strategies with marker-based encoding of neural networks," Dept. Comput. Sci., Univ. Texas at Austin, Austin, TX, 1993. Tech. Rep. AI93-206.
64. D. E. Moriarty, "Symbiotic evolution of neural networks in sequential decision tasks," Ph.D. dissertation, Dept. Comput. Sci., Univ. Texas at Austin, Austin, TX, 1997. Tech. Rep. UT-A197-257.
65. A. Agogino, K. O. Stanley, and R. Miikkulainen, "Real-time interactive neuro-evolution," Neural Process. Lett., vol. 11, pp. 29-38, 2000.
66. D. Moriarty and R. Miikkulainen, "Learning sequential decision tasks," Dept. Comput. Sci., Univ. Texas at Austin, Austin, TX, 1995. Tech. Rep. A195-229.
67. D. E. Moriarty and R. Miikkulainen, "Discovering complex Othello strategies through evolutionary neural networks," Connection Sci., vol. 7, no. 3, pp. 195-209, 1995.
68. N. J. Radcliffe, "Genetic set recombination and its application to neural network topology optimization;'Neural Comput. Appl., vol. 1, no. 1, pp. 67-90, 1993.
69. A. H. Wright, "Genetic algorithms for real parameter optimization," in Foundations of Genetic Algorithms, G. J. E. Rawlins, Ed. San Mateo, CA: Morgan Kaufmann, 1991, pp. 205-218.
70. D. E. Goldberg and J. Richardson, "Genetic algorithms with sharing for multimodal function optimization," in Proc. 2nd Int. Conf. Genetic Algorithms, J. J. Grefenstette, Ed., 1987, pp. 148-154.
71. Handbook of Evolutionary Computation, Oxford Univ. Press, London, U.K., 1995. W. Spears, "Speciation using tag bits".
72. A. Amores, A. Force, Y.-L. Yan, L. Joly, C. Amemiya, A. Fritz, R. K. Ho, J. Langeland, V. Prince, Y.-L. Wang, M. Westerfield, M. Ekker, and J. H. Postlethwait, "Zebrafish HOX clusters and vertebrate genome evolution," Science, vol. 282, pp. 1711-1784, 1998.
73. S. B. Carroll, "Homeotic genes and the evolution of arthropods and chordates," Nature, vol. 376, pp. 479-485, 1995.
74. A. Force, M. Lynch, F. B. Pickett, A. Amores, Y. Lin Yan, and J. Postlethwait, "Preservation of duplicate genes by complementary, degenerative mutations," Genetics, vol. 151, pp. 1531-1545, 1999.
75. A. P. Martin, "Increasing genomic complexity by gene duplication and the origin of vertebrates," The Amer. Naturalist, vol. 154, no. 2, pp. 111-128, 1999.
76. K. O. Stanley and R. Miikkulainen, "Efficient reinforcement learning through evolving neural network topologies," in Proc. Genetic Evol. Comput. Conf., 2002, pp. 569-577.
77. H.-G. Beyer and H. P. Schwefel, "Evolution strategies - A comprehensive introduction," Natural Computing, vol. 1, no. 1, pp. 3-52, 2002. [Online]. Available: http://www.kluweronline.com/ article.asp?PIPS=401777.