Mastering the game of Go without human knowledge

Nature

Volumn 550, Issue 7676, 2017, Pages 354-359

  Silver, David ^a   Schrittwieser, Julian ^a   Simonyan, Karen ^a   Antonoglou, Ioannis ^a   Huang, Aja ^a   Guez, Arthur ^a   Hubert, Thomas ^a   Baker, Lucas ^a   Lai, Matthew ^a   Bolton, Adrian ^a   Chen, Yutian ^a   Lillicrap, Timothy ^a   Hui, Fan ^a   Sifre, Laurent ^a   Van Den Driessche, George ^a   Graepel, Thore ^a   Hassabis, Demis ^a  

^a DEEPMIND   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHM; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; KNOWLEDGE; SUPERVISED LEARNING;

ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; GAME; KNOWLEDGE; LEARNING ALGORITHM; PRIORITY JOURNAL; REINFORCEMENT; HUMAN; RECREATIONAL GAME; SOFTWARE; SUPERVISED MACHINE LEARNING; UNSUPERVISED MACHINE LEARNING;

GAMES, RECREATIONAL; HUMANS; NEURAL NETWORKS (COMPUTER); REINFORCEMENT (PSYCHOLOGY); SOFTWARE; SUPERVISED MACHINE LEARNING; UNSUPERVISED MACHINE LEARNING;

EID: 85031918331     PISSN: 00280836     EISSN: 14764687     Source Type: Journal    
DOI: 10.1038/nature24270     Document Type: Article

References (69)

1. Friedman, J., Hastie, T. & Tibshirani, R. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Springer, 2009).
2. LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436-444 (2015).
3. Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. In Adv. Neural Inf. Process. Syst. Vol. 25 (eds Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097-1105 (2012).
4. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. 29th IEEE Conf. Comput. Vis. Pattern Recognit. 770-778 (2016).
5. Hayes-Roth, F., Waterman, D. & Lenat, D. Building Expert Systems (Addison-Wesley, 1984).
6. Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529-533 (2015).
7. Guo, X., Singh, S. P., Lee, H., Lewis, R. L. & Wang, X. Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In Adv. Neural Inf. Process. Syst. Vol. 27 (eds Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N. D. & Weinberger, K. Q.) 3338-3346 (2014).
8. Mnih, V. et al. Asynchronous methods for deep reinforcement learning. In Proc. 33rd Int. Conf. Mach. Learn. Vol. 48 (eds Balcan, M. F. & Weinberger, K. Q.) 1928-1937 (2016).
9. Jaderberg, M. et al. Reinforcement learning with unsupervised auxiliary tasks. In 5th Int. Conf. Learn. Representations (2017).
10. Dosovitskiy, A. & Koltun, V. Learning to act by predicting the future. In 5th Int. Conf. Learn. Representations (2017).
11. Mańdziuk, J. in Challenges for Computational Intelligence (Duch, W. & Mańdziuk, J.) 407-442 (Springer, 2007).
12. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484-489 (2016).
13. Coulom, R. Efficient selectivity and backup operators in Monte-Carlo tree search. In 5th Int. Conf. Computers and Games (eds Ciancarini, P. & van den Herik, H. J.) 72-83 (2006).
14. Kocsis, L. & Szepesvári, C. Bandit based Monte-Carlo planning. In 15th Eu. Conf. Mach. Learn. 282-293 (2006).
15. Browne, C. et al. A survey of Monte Carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 4, 1-49 (2012).
16. Fukushima, K. Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36, 193-202 (1980).
17. LeCun, Y. & Bengio, Y. in The Handbook of Brain Theory and Neural Networks Ch. 3 (ed. Arbib, M.) 276-278 (MIT Press, 1995).
18. Ioffe, S. & Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proc. 32nd Int. Conf. Mach. Learn. Vol. 37 448-456 (2015).
19. Hahnloser, R. H. R., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J. & Seung, H. S. Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405, 947-951 (2000).
20. Howard, R. Dynamic Programming and Markov Processes (MIT Press, 1960).
21. Sutton, R. & Barto, A. Reinforcement Learning: an Introduction (MIT Press, 1998).
22. Bertsekas, D. P. Approximate policy iteration: a survey and some new methods. J. Control Theory Appl. 9, 310-335 (2011).
23. Scherrer, B. Approximate policy iteration schemes: a comparison. In Proc. 31st Int. Conf. Mach. Learn. Vol. 32 1314-1322 (2014).
24. Rosin, C. D. Multi-armed bandits with episode context. Ann. Math. Artif. Intell. 61, 203-230 (2011).
25. Coulom, R. Whole-history rating: a Bayesian rating system for players of time-varying strength. In Int. Conf. Comput. Games (eds van den Herik, H. J., Xu, X. Ma, Z. & Winands, M. H. M.) Vol. 5131 113-124 (Springer, 2008).
26. Laurent, G. J., Matignon, L. & Le Fort-Piat, N. The world of independent learners is not Markovian. Int. J. Knowledge-Based Intelligent Engineering Systems 15, 55-64 (2011).
27. Foerster, J. N. et al. Stabilising experience replay for deep multi-agent reinforcement learning. In Proc. 34th Int. Conf. Mach. Learn. Vol. 70 1146-1155 (2017).
28. Heinrich, J. & Silver, D. Deep reinforcement learning from self-play in imperfect-information games. In NIPS Deep Reinforcement Learning Workshop (2016).
29. Jouppi, N. P. et al. In-datacenter performance analysis of a Tensor Processing Unit. Proc. 44th Annu. Int. Symp. Comp. Architecture Vol. 17 1-12 (2017).
30. Maddison, C. J., Huang, A., Sutskever, I. & Silver, D. Move evaluation in Go using deep convolutional neural networks. In 3rd Int. Conf. Learn. Representations. (2015).
31. Clark, C. & Storkey, A. J. Training deep convolutional neural networks to play Go. In Proc. 32nd Int. Conf. Mach. Learn. Vol. 37 1766-1774 (2015).
32. Tian, Y. & Zhu, Y. Better computer Go player with neural network and long-term prediction. In 4th Int. Conf. Learn. Representations (2016).
33. Cazenave, T. Residual networks for computer Go. IEEE Trans. Comput. Intell. AI Games https://doi.org/10.1109/TCIAIG.2017.2681042 (2017).
34. Huang, A. AlphaGo master online series of games. https://deepmind.com/ research/AlphaGo/match-archive/master (2017).
35. Barto, A. G. & Duff, M. Monte Carlo matrix inversion and reinforcement learning. Adv. Neural Inf. Process. Syst. 6, 687-694 (1994).
36. Singh, S. P. & Sutton, R. S. Reinforcement learning with replacing eligibility traces. Mach. Learn. 22, 123-158 (1996).
37. Lagoudakis, M. G. & Parr, R. Reinforcement learning as classification: leveraging modern classifiers. In Proc. 20th Int. Conf. Mach. Learn. 424-431 (2003).
38. Scherrer, B., Ghavamzadeh, M., Gabillon, V., Lesner, B. & Geist, M. Approximate modified policy iteration and its application to the game of Tetris. J. Mach. Learn. Res. 16, 1629-1676 (2015).
39. Littman, M. L. Markov games as a framework for multi-agent reinforcement learning. In Proc. 11th Int. Conf. Mach. Learn. 157-163 (1994).
40. Enzenberger, M. The integration of a priori knowledge into a Go playing neural network. http://www.cgl.ucsf.edu/go/Programs/neurogo-html/neurogo.html (1996).
41. Enzenberger, M. in Advances in Computer Games (eds Van Den Herik, H. J., Iida, H. & Heinz, E. A.) 97-108 (2003).
42. Sutton, R. Learning to predict by the method of temporal differences. Mach. Learn. 3, 9-44 (1988).
43. Schraudolph, N. N., Dayan, P. & Sejnowski, T. J. Temporal difference learning of position evaluation in the game of Go. Adv. Neural Inf. Process. Syst. 6, 817-824 (1994).
44. Silver, D., Sutton, R. & Müller, M. Temporal-difference search in computer Go. Mach. Learn. 87, 183-219 (2012).
45. Silver, D. Reinforcement Learning and Simulation-Based Search in Computer Go. PhD thesis, Univ. Alberta, Edmonton, Canada (2009).
46. Gelly, S. & Silver, D. Monte-Carlo tree search and rapid action value estimation in computer Go. Artif. Intell. 175, 1856-1875 (2011).
47. Coulom, R. Computing Elo ratings of move patterns in the game of Go. Int. Comput. Games Assoc. J. 30, 198-208 (2007).
48. Gelly, S., Wang, Y., Munos, R. & Teytaud, O. Modification of UCT with patterns in Monte-Carlo Go. Report No. 6062 (INRIA, 2006).
49. Baxter, J., Tridgell, A. & Weaver, L. Learning to play chess using temporal differences. Mach. Learn. 40, 243-263 (2000).
50. Veness, J., Silver, D., Blair, A. & Uther, W. Bootstrapping from game tree search. In Adv. Neural Inf. Process. Syst. 1937-1945 (2009).
51. Lai, M. Giraffe: Using Deep Reinforcement Learning to Play Chess. MSc thesis, Imperial College London (2015).
52. Schaeffer, J., Hlynka, M. & Jussila, V. Temporal difference learning applied to a high-performance game-playing program. In Proc. 17th Int. Jt Conf. Artif. Intell. Vol. 1 529-534 (2001).
53. Tesauro, G. TD-gammon, a self-teaching backgammon program, achieves master-level play. Neural Comput. 6, 215-219 (1994).
54. Buro, M. From simple features to sophisticated evaluation functions. In Proc. 1st Int. Conf. Comput. Games 126-145 (1999).
55. Sheppard, B. World-championship-caliber Scrabble. Artif. Intell. 134, 241-275 (2002).
56. Moravčík, M. et al. DeepStack: expert-level artificial intelligence in heads-up no-limit poker. Science 356, 508-513 (2017).
57. Tesauro, G & Galperin, G. On-line policy improvement using Monte-Carlo search. In Adv. Neural Inf. Process. Syst. 1068-1074 (1996).
58. Tesauro, G. Neurogammon: a neural-network backgammon program. In Proc. Int. Jt Conf. Neural Netw. Vol. 3, 33-39 (1990).
59. Samuel, A. L. Some studies in machine learning using the game of checkers II-recent progress. IBM J. Res. Develop. 11, 601-617 (1967).
60. Kober, J., Bagnell, J. A. & Peters, J. Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32, 1238-1274 (2013).
61. Zhang, W. & Dietterich, T. G. A reinforcement learning approach to job-shop scheduling. In Proc. 14th Int. Jt Conf. Artif. Intell. 1114-1120 (1995).
62. Cazenave, T., Balbo, F. & Pinson, S. Using a Monte-Carlo approach for bus regulation. In Int. IEEE Conf. Intell. Transport. Syst. 1-6 (2009).
63. Evans, R. & Gao, J. Deepmind AI reduces Google data centre cooling bill by 40%. https://deepmind.com/blog/deepmind-ai-reduces-google-data-centrecooling-bill-40/ (2016).
64. Abe, N. et al. Empirical comparison of various reinforcement learning strategies for sequential targeted marketing. In IEEE Int. Conf. Data Mining 3-10 (2002).
65. Silver, D., Newnham, L., Barker, D., Weller, S. & McFall, J. Concurrent reinforcement learning from customer interactions. In Proc. 30th Int. Conf. Mach. Learn. Vol. 28 924-932 (2013).
66. Tromp, J. Tromp-Taylor rules. http://tromp.github.io/go.html (1995).
67. Müller, M. Computer Go. Artif. Intell. 134, 145-179 (2002).
68. Shahriari, B., Swersky, K., Wang, Z., Adams, R. P. & de Freitas, N. Taking the human out of the loop: a review of Bayesian optimization. Proc. IEEE 104, 148-175 (2016).
69. Segal, R. B. On the scalability of parallel UCT. Comput. Games 6515, 36-47 (2011).