Reproducing Neural Network Research Findings via Reverse Engineering: Replication of AlphaGo Zero by Crowdsourced Leela Zero
Abstract
The reproducibility of scientific findings is essential to the integrity of research. The scientific method requires hypotheses to be validated independently by different laboratories. Investigators are expected to provide sufficient information in their publications to permit an objective evaluation of their methods and an independent reproduction of their results. This is particularly true for research supported by public funds, where transparency of both methods and findings represents a return on public investment. Unfortunately, many publications fall short of this standard for various reasons, including a desire to protect intellectual property or national security. The reproducibility of findings is essential in transferring machine learning findings from research into healthcare practice. Fortunately, the internet makes it easier to overcome these limitations by permitting multiple individuals to participate in reproducibility efforts and to crowdsource the reverse engineering of novel software. We present a case study of this capability from neural network research. The success of the crowdsourced project Leela Zero to reverse engineer the findings of AlphaGo Zero exemplifies the ability to reproduce novel results despite the lack of extensive computational resources or a detailed description of the initial experimental methods. The implications of this successful reverse engineering effort for future reproducibility of neural network research are discussed.
Downloads
Metrics
PlumX Statistics
References
2. Allis, L.V., Herik, H. J., & Herschberg, I. S. (1990). Which Games Will Survive? Retrieved from https://books.google.com/books?id=PlZAGwAACAAJ
3. Baker, M. (2016). 1,500 scientists lift the lid on reproducibility. Nature News, 533(7604), 452. https://doi.org/10.1038/533452a
4. Chess.com (2018). Contributing to Leela Chess Zero. Creating the Caissa of Chess engines. Retrieved August 19, 2019, from https://www.chess.com/forum/view/general/contributing-to-leela-chess-zero-creating-the-caissa-of-chess-engines
5. CBInsights. (2019). China Is Starting To Edge Out The U.S. In A.I. Investment—C.B. Insights Research. (2019, February 12). Retrieved August 6, 2019, from https://www.cbinsights.com/research/china-artificial- intelligence-investment-startups-tech/
6. Chess.com (2018). Contributing to Leela Chess Zero. Creating the Caissa of Chess engines. Retrieved August 19, 2019, from https://www.chess.com/forum/view/general/contributing-to-leela-chess-zero-creating-the-caissa-of-chess-engines
7. Colaboratory (n.d.). Frequently asked questions. Retrieved August 19, 2019, from https://research.google.com/colaboratory/faq.html
8. Go ranks and ratings (n.d.) Wikipedia. Retrieved as https://en.wikipedia.org/wiki/Go_ranks_and_ratings.
9. Greene, C. S., Garmire, L. X., Gilbert, J. A., Ritchie, M. D., & Hunter, L. E. (2017). Celebrating parasites. Nature Genetics, 49(4), 483–484. https://doi.org/10.1038/ng.3830
10. Kharpal A. (2017). China wants to be a $150 billion world leader in A.I. by 2030. (2017, July 21). Retrieved August 6, 2019, from https://www.cnbc.com/2017/07/21/china-ai-world-leader-by-2030.html
11. Koch C (2016). How the Computer Beat the Go Master. Retrieved at https://www.scientificamerican.com/article/how-the-computer-beat-the-go-master/
12. Krikke, J. (2007). The Challenge of Go. IEEE Intelligent Systems, 22(01), 7–8. https://doi.org/10.1109/5254.747899
13. LCZero. (n.d.). Retrieved February 23, 2019, from http://lczero.org/
14. LCZero (2019). Leela won the TCEC CUP! Retrieved February 23, 2019, from http://blog.lczero.org/2019/02/leela-won-tcec-cup.html
15. Levinovitz, A. (2014). The Mystery of Go, the Ancient Game That Computers Still Can't Win. Retrieved at https://www.wired.com/2014/05/the-world-of-computer-go/
16. Lystad, T.A. (2018) GM Andrew Tang Defends Humanity Against Leela Chess Zero. Retrieved August 19, 2019, from Lichess.org website: https://lichess.org/blog/WtzZAyoAALvE8ZSQ/gm-andrew-tang- defends-humanity-against-Leela-chess-zero
17. Mozur, P. (2017) Beijing Wants A.I. to Be Made in China by 2030. The New York Times. (2017, July 20). Retrieved August 6, 2019, from https://www.nytimes.com/2017/07/20/business/china-artificial-intelligence.html
18. National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and replicability in science. National Academies Press. https://doi.org/10.17226/25303
19. Pascutto GC. (2017). [Computer-go] Zero performance. Retrieved at http://computer-go.org/pipermail/computer-go/2017-October/010307.html
20. Pascutto GC (2018). Gian-Carlo Pascutto - The man behind LeelaZero. Retrieved at https://www.eurogofed.org/?id=205
21. Sadler, M., & Regan, N. (2019). Game Changer: AlphaZero's Chess Strategies and the Promise of A.I. Retrieved from https://books.google.com/books?id=KhhivwEACAAJ
22. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., ... & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. nature, 529(7587), 484-489. Retrieved from http://arxiv.org/abs/1712.01815
23. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T. and Lillicrap, T., (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815.
24. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Bolton, A. (2017b). Mastering the game of Go without human knowledge. Nature, 550(7676), 354.
25. Silver, D., Sutton, R. S., & Müller, M. (2007, January). Reinforcement Learning of Local Shape in the Game of Go. In IJCAI (Vol. 7, pp. 1053-1058)
26. Tian, Y. & Zitnik, K. (2018). Facebook Open Sources ELF OpenGo. Retrieved at https://research.facebook.com/blog/2018/05/facebook-open-sources-elf-opengo/.
27. Zaman, R., Prokhorov, D., & Wunsch, D. C. (1997). Adaptive critic design in learning to play the game of Go. Proceedings of International Conference on Neural Networks (ICNN'97), 1,ICNN'97l.1. https://doi.org/10.1109/ICNN.1997.611623
28. Zaman, R., & Wunsch, D. C. (1999). T.D. Methods applied to the mixture of experts for learning 9/spl times/9 Go evaluation function. IJCNN'99. InIJCNN'99nal Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339), 6, 3734–3739 vol.6. https://doi.org/10.1109/IJCNN.1999.830746
29. Zastrow, M. (2016). South Korea trumpets $860-million AI fund after AlphaGo 'shock'. Nature. https://doi.org/10.1038/nature.2016.19595
30. Zero (n.d.) Retrieved at https://zero.sjeng.org/
31. Zhe, L. (2016) [Game 1] Lee Sedol’s Strategy and AlphaGo’s weakness. Retrieved at
https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-AlphaGos-weakness/comment-page-1/#comment-25
Copyright (c) 2022 Dustin Tanksley, Daniel B. Hier, Donald C. Wunsch II
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
.jpg)
