Coevolution versus self-play temporal difference learning for acquiring position evaluation in small-board go

IEEE Transactions on Evolutionary Computation

Volumn 9, Issue 6, 2005, Pages 628-640

  Runarsson, Thomas Philip ^a   Lucas, Simon M ^b  

^a UNIVERSITY OF ICELAND   (Iceland)

^b UNIVERSITY OF ESSEX   (United Kingdom)

Author keywords

Coevolution; Evolution strategy; Game Strategies; Reinforcement learning; Temporal difference learning

Indexed keywords

EVOLUTIONARY ALGORITHMS; GAME THEORY; GRADIENT METHODS; LEARNING ALGORITHMS; OPTIMIZATION;

COEVOLUTION; EVOLUTION STRATEGY; GAME STRATEGIES; REINFORCEMENT LEARNING; TEMPORAL DIFFERENCE LEARNING;

LEARNING SYSTEMS;

EID: 29244474089     PISSN: 1089778X     EISSN: None     Source Type: Journal    
DOI: 10.1109/TEVC.2005.856212     Document Type: Article

References (46)

1. C. Shannon, "Programming a computer for playing chess, Philos. Mag., vol. 41, pp. 256-275, 1950.
2. A. Samuel, "Some studies in machine learning using the game of checkers," IBM J. Res. Develop., vol. 3, pp. 211-229, 1959.
3. D. Michie, "Trial and error," in Science Survey, Part 2. Baltimore,MD: Penguin, 1961, pp. 129-145.
4. J. Pollack and A. Blair, "Co-evolution in the successful learning of backgammon strategy," Mach. Learn., vol. 32, pp. 225-240, 1998.
5. K. Chellapilla and D. Fogel, "Evolving neural networks to play checkers without expert knowledge," IEEE Trans. Neural Netw., vol. 10, no. 6, pp. 1382-1391, 1999.
6. K. Chellapilla and D. Fogel, "Evolving an expert checkers playing program without using human expertise," IEEE Trans. Evol. Comput., vol. 5, no. 5, pp. 422-428, Oct. 2001.
7. D. Fogel, Blondie24: playing at the edge of AI. San Mateo, CA: Morgan Kaufmann, 2002.
8. C. Kotnik and J. Kalita, "The significance of temporal-difference learning in self-play training: TD-rummy versus EVO-rummy," in Proc. Int. Conf. Mach. Learn. (ICML), 2003, pp. 369-375.
9. P. J. Darwen, "Why co-evolution beats temporal difference learning at backgammon for a linear architecture, but not a nonlinear architecture;' in Proc. Congr. Evol. Comput., 2001, pp. 1003-1010. [Online]. Available: citeseer.ist.psu.edu/article/darwenO1why.html.
10. G. Tesauro, "Comments on 'co-evolution in the successful learning of backgammon strategy'" Mach. Learn., vol. 32, no. 3, pp. 241-243, 1998.
11. G. Tesauro, "Temporal difference learning and TD-gammon," Commun. ACM, vol. 38, no. 3, pp. 58-68, 1995.
12. K. Stanley and R. Miikkulainen, "Evolving a roving eye for go," in Proc. Genetic Evol. Comput. Conf., 2004, pp. 1226-1238.
13. N. Richards, D. Moriarty, and R. Miikkulainen, "Evolving neural networks to play go, Appl. Intell., vol. 8, pp. 85-96, 1998.
14. D. Fogel, T. Hays, S. Hahn, and J. Quon, "An evolutionary self-learning chess program," Proc. IEEE, pp. 1947-1954, Dec. 2004.
15. K.-J. Kim and S.-B. Cho, "Evolving speciated checkers players with crowding algorithm," in Proc. Congr. Evolut. Comput., vol. 1, 2002, pp. 407-412.
16. D. Moriarty, A. Schultz, and J. Grefenstette, "Reinforcement learning through evolutionary computation," J. Artif. Intell. Res., vol. 11, pp. 241-276, 1999.
17. N. Franken and A. Engelbrecht, "Comparing PSO structures to learn the game of checkers from zero knowledge," in Proc. Congr. Evol. Comput., 2003, pp. 234-241.
18. L. Messerschmidt and A. Engelbrecht, "Learning to play games using a PSO-based competitive learning approach," IEEE Trans. Evol. Comput., vol. 8, no. 3, pp. 280-288, Jun. 2004.
19. R. Sutton and A. Barto, Introduction to Reinforcement Learning. Cambridge, MA: MIT Press, 1998.
20. D. Fogel, "Using evolutionary programming to create networks that are capable of playing tic-tac-toe," in Proc. IEEE Int. Conf. Neural Netw., 1993, pp. 875-880.
21. P. J. Angeline and J. B. Pollack, "Competitive environments evolve better solutions for complex tasks," in Proc. 5th Int. Conf. Genetic Algorithms, S. Forrest, Ed., 1993, pp. 264-270.
22. T. Graepel, M. Goutrie, M. Krüger, and R. Herbrich, "Learning on graphs in the game of Go," in Proc. Int. Conf. Artif. Neural Netw., 2001, pp. 347-352.
23. N. Schraudolph, P. Dayan, and T. Sejnowski, "Temporal difference learning of position evaluation in the game of Go," in Proc. Neural Inf. Process. Syst. 6, 1994, pp. 817-824.
24. M. Enzenberger, "Evaluation in Go by a neural network using soft segmentation," in Proc. 10th Advances Comput. Games Conf., 2003, pp. 97-108.
25. G. Kendall, R. Yaakob, and P. Hingston, "An investigation of an evolutionary approach of the opening of Go," in Proc. Congr. Evol. Comput., 2004, pp. 2052-2059.
26. D. Fotland. (1993) Knowledge representation in the many faces of Go. Tech. Rep., American Go Association. [Online]. Available: ftp:// bsd-server.ucsfedu/Go/comp/mfg.Z
27. M. Muller, "Computer go," Artif. Intell., vol. 134, pp. 145-179, 2002.
28. M. Enzenberger. (1996) The integration of a priori knowledge into a Go playing neural network. [Online]. Available: http://www. markus-enzenberger.de/neurogol996.html
29. S. Ficici, "Solution Concepts in Coevolutionary Algorithms," Ph.D. dissertation, Brandeis Univ., Waltham, MA, 2004.
30. D. Hillis, "Co-evolving parasites improves simulated evolution as an optimization procedure," Physica D, vol. 42, pp. 228-234, 1990.
31. A. Lubberts and R. Miikkulainen, "Co-evolving a Go-playing neural network," in Proc. Genetic Evol. Comput. Conf. Workshop Program, 2001, pp. 14-19.
32. R. Lippmann, "An introduction to computing with neural nets," IEEE ASSP Mag., pp. 4-22, Apr. 1987.
33. E. Hughes, "Piece difference: Simple to evolve?," in Proc. Congr. Evol. Comput., 2003, pp. 2470-2473.
34. K. Stanley and R. Miikulainen, "Competitive coevolution through evolutionary complexification," J. Artificial Intell. Res., vol. 21, pp. 63-100, 2004.
35. H. Kitano, "Designing neural networks using genetic algorithm with graph generation system," Complex Syst., vol. 4, pp. 461-476, 1990.
36. F. Gruau, "Cellular encoding of genetic neural networks," Ecole Normale Superieure de Lyon, Laboratoire de l'Informatique du Parallelisme, Tech. Rep. 92-21, 1992.
37. F. Gruau, D. Whitley, and L. Pyeatt, "A comparison between cellular encoding and direct encoding for genetic neural networks," Neuro-colt Tech. Rep. NC-TR-96-048, 1996.
38. J. Schaeffer, M. Hlynka, and V. Jussila, "Temporal difference learning applied to a high-performance game-playing program," in Proc. Int. Joint Conf. Artif. Intell., 2001, pp. 529-534.
39. E. Ven Der Werf, J. Ven Den Herik, and J. Uiterwijk, "Solving go on small boards," ICGA J., vol. 26, no. 2, pp. 92-107, 2003.
40. E. Van Der Werf, "AI techniques for the game of Go," Ph.D. dissertation, Maastricht Univ., Maastricht, The Netherlands, 2005.
41. B. Bouzy and I Cazenave, "Computer Go: An AI-oriented survey," Artif. Intell. J., pp. 39-103, 2001.
42. J. Baxter, A. Trigdell, and L. Weaver, "Knightcap: A chess program that learns by combining TD(λ) with game-tree search," in Proc. 15th Int. Conf. Mach. Learn., 1998, pp. 28-36. [Online]. Available: citeseer.ist.psu.edu/baxter98knightcap.html.
43. I. Rechenberg, Evolutionstrategie'94. Stuttgart, Germany: Frommann-Holzboog, 1994.
44. T. P. Runarsson, "Reducing random fluctuations in mutative self-adaptation," in Lecture Notes in Computer Science. Berlin, Germany: Springer-Verlag, 2002, vol. 2439, Parallel Problem Solving from Nature VII (PPSN-2002), pp. 194-203.
45. X. Yao, "Evolving artificial neural networks,"Proc. IEEE, vol. 87, no. 9, pp. 1423-1447, 1999.
46. S. Lucas, "Forward-backward building blocks for evolving neural networks with intrinsic learning behaviors," in Lecture Notes in Computer Science (1240): Biological and Artificial Computation: From Neuro-science to Technology. Berlin, Germany: Springer-Verlag, 1997, pp. 723-732.