Reproducing Neural Network Research Findings via Reverse Engineering: Replication of AlphaGo Zero by Crowdsourced Leela Zero
Abstract
The reproducibility of scientific findings is essential to the integrity of research. The scientific method requires hypotheses to be validated independently by different laboratories. Investigators are expected to provide sufficient information in their publications to permit an objective evaluation of their methods and an independent reproduction of their results. This is particularly true for research supported by public funds, where transparency of both methods and findings represents a return on public investment. Unfortunately, many publications fall short of this standard for various reasons, including a desire to protect intellectual property or national security. The reproducibility of findings is essential in transferring machine learning findings from research into healthcare practice. Fortunately, the internet makes it easier to overcome these limitations by permitting multiple individuals to participate in reproducibility efforts and to crowdsource the reverse engineering of novel software. We present a case study of this capability from neural network research. The success of the crowdsourced project Leela Zero to reverse engineer the findings of AlphaGo Zero exemplifies the ability to reproduce novel results despite the lack of extensive computational resources or a detailed description of the initial experimental methods. The implications of this successful reverse engineering effort for future reproducibility of neural network research are discussed.
Downloads
PlumX Statistics
References
Allis, L. Victor. (1994). Searching for solutions in games and artificial intelligence.
Allis, L.V., Herik, H. J., & Herschberg, I. S. (1990). Which Games Will Survive? Retrieved from https://books.google.com/books?id=PlZAGwAACAAJ
Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452. https://doi.org/10.1038/533452a
Chess.com (2018). Contributing to Leela Chess Zero. Creating the Caissa of Chess engines. Retrieved August 19, 2019, from https://www.chess.com/forum/view/general/contributing-to-leela-chess-zero-creating-the-caissa-of-chess-engines
CBInsights. (2019). China Is Starting To Edge Out The U.S. In A.I. Investment—C.B. Insights Research. (2019, February 12). Retrieved August 6, 2019, from https://www.cbinsights.com/research/china-artificial- intelligence-investment-startups-tech/
Chess.com (2018). Contributing to Leela Chess Zero. Creating the Caissa of Chess engines. Retrieved August 19, 2019, from https://www.chess.com/forum/view/general/contributing-to-leela-chess-zero-creating-the-caissa-of-chess-engines
Colaboratory (n.d.). Frequently asked questions. Retrieved August 19, 2019, from https://research.google.com/colaboratory/faq.html
Go ranks and ratings (n.d.) Wikipedia. Retrieved as https://en.wikipedia.org/wiki/Go_ranks_and_ratings.
Greene, C. S., Garmire, L. X., Gilbert, J. A., Ritchie, M. D., & Hunter, L. E. (2017). Celebrating parasites. Nature Genetics, 49(4), 483–484. https://doi.org/10.1038/ng.3830
Kharpal A. (2017). China wants to be a $150 billion world leader in A.I. by 2030. (2017, July 21). Retrieved August 6, 2019, from https://www.cnbc.com/2017/07/21/china-ai-world-leader-by-2030.html
Koch C (2016). How the Computer Beat the Go Master. Retrieved at https://www.scientificamerican.com/article/how-the-computer-beat-the-go-master/
Krikke, J. (2007). The Challenge of Go. IEEE Intelligent Systems, 22(01), 7–8. https://doi.org/10.1109/5254.747899
LCZero. (n.d.). Retrieved February 23, 2019, from http://lczero.org/
LCZero (2019). Leela won the TCEC CUP! Retrieved February 23, 2019, from http://blog.lczero.org/2019/02/leela-won-tcec-cup.html
Levinovitz, A. (2014). The Mystery of Go, the Ancient Game That Computers Still Can't Win. Retrieved at https://www.wired.com/2014/05/the-world-of-computer-go/
Lystad, T.A. (2018) GM Andrew Tang Defends Humanity Against Leela Chess Zero. Retrieved August 19, 2019, from Lichess.org website: https://lichess.org/blog/WtzZAyoAALvE8ZSQ/gm-andrew-tang- defends-humanity-against-Leela-chess-zero
Mozur, P. (2017) Beijing Wants A.I. to Be Made in China by 2030. The New York Times. (2017, July 20). Retrieved August 6, 2019, from https://www.nytimes.com/2017/07/20/business/china-artificial-intelligence.html
National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and replicability in science. National Academies Press. https://doi.org/10.17226/25303
Pascutto GC. (2017). [Computer-go] Zero performance. Retrieved at http://computer-go.org/pipermail/computer-go/2017-October/010307.html
Pascutto GC (2018). Gian-Carlo Pascutto - The man behind LeelaZero. Retrieved at https://www.eurogofed.org/?id=205
Sadler, M., & Regan, N. (2019). Game Changer: AlphaZero's Chess Strategies and the Promise of A.I. Retrieved from https://books.google.com/books?id=KhhivwEACAAJ
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. nature, 529(7587), 484-489. Retrieved from http://arxiv.org/abs/1712.01815
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T. and Lillicrap, T., (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Bolton, A. (2017b). Mastering the game of Go without human knowledge. Nature, 550(7676), 354.
Silver, D., Sutton, R. S., & Müller, M. (2007, January). Reinforcement Learning of Local Shape in the Game of Go. In IJCAI (Vol. 7, pp. 1053-1058)
Tian, Y. & Zitnik, K. (2018). Facebook Open Sources ELF OpenGo. Retrieved at https://research.facebook.com/blog/2018/05/facebook-open-sources-elf-opengo/.
Zaman, R., Prokhorov, D., & Wunsch, D. C. (1997). Adaptive critic design in learning to play the game of Go. Proceedings of International Conference on Neural Networks (ICNN'97), 1,ICNN'97l.1. https://doi.org/10.1109/ICNN.1997.611623
Zaman, R., & Wunsch, D. C. (1999). T.D. Methods applied to the mixture of experts for learning 9/spl times/9 Go evaluation function. IJCNN'99. InIJCNN'99nal Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339), 6, 3734–3739 vol.6. https://doi.org/10.1109/IJCNN.1999.830746
Zastrow, M. (2016). South Korea trumpets $860-million AI fund after AlphaGo 'shock'. Nature. https://doi.org/10.1038/nature.2016.19595
Zero (n.d.) Retrieved at https://zero.sjeng.org/
Zhe, L. (2016) [Game 1] Lee Sedol’s Strategy and AlphaGo’s weakness. Retrieved at
Copyright (c) 2022 Dustin Tanksley, Daniel B. Hier, Donald C. Wunsch II
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
