Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

By David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel and 3 more. 2017

2,110 citations recorded by the source corpus.

The game of chess is the most widely-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. In contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go, by tabula rasa reinforcement learning from games of self-play. In this paper, we generalise this approach into a single AlphaZero algorithm that can achieve, tabula rasa, superhuman performance in many challenging domains. Starting from random play, and given no domain knowledge except the game rules, AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi (Japanese chess) as well as Go, and convincingly defeated a world-champion program in each case.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, background): “• AI systems can already perform some tasks at or beyond human level (e.g. [Silver et al., 2017]), and over time more examples are likely to emerge.”
AI safety via debate (2018, methodology): “For example, the ﬁrst AlphaGo result [Silver et al., 2016] used an opponent pool but later versions do not [Silver et al., 2017a,b].”
Adversarial Training for High-Stakes Reliability (2022, background): “Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1, 2, 3, 4], robotics [5, 6], and language modeling and text-based reasoning [7, 8, 9, 10, 11].”
Generative Language Modeling for Automated Theorem Proving (2020, background): “With the possible exception of AlphaGo [13] and AlphaZero [23], reasoning tasks are conspicuously absent from the list above.”
Green AI (2019, background): “1 Figure 1 reproduced from [2] plots training cost increase over time for state-of-the-art deep learning models starting with AlexNet in 2012 [20] to AlphaZero in 2017 [37].”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “Recently AlphaGo Zero [9] surpassed its previous version without using human knowledge and a generic version, AlphaZero [10], achieved a superhuman level within 24 hours of cross domains of chess, Shogi, and Go.”
Benchmark and Survey of Automated Machine Learning Frameworks (2019, background): “Self-play (Lake et al., 2017) is a reinforcement learning strategy that has received a lot of attention lately due to the recent successes of AlphaZero (Silver et al., 2017).”
Survey on Automated Machine Learning (2019, background): “, AlphaGO (Silver et al., 2017) was able to beat the human champion in GO.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “methods like temporal difference (TD), Monte Carlo (MC), and Monte Carlo Tree Search (MCTS) take a forward view approach2 by estimating the expected future return [115]. MCTS is the core of AlphaZero [105, 106], which learned to play Chess and Go better than human professionals. The future return is”
On the Binding Problem in Artificial Neural Networks (2020, background): “In reinforcement learning, where the use of neural networks has led to superhuman performance in gameplay (Mnih et al., 2015; Silver et al., 2017; Berner et al., 2019), it is found that agents are fragile under distributional shift (Kansky et al.”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “In similar spirit to our ADR, related works include PowerPlay [97, 103], POET [117] and different varieties of self-play [101, 102, 76].”
Programmatically Interpretable Reinforcement Learning (2018, background): “Deep reinforcement learning (D RL ) has had a massive impact on the ﬁeld of machine learning and has led to remarkable successes in the solution of many challenging tasks (Mnih et al., 2015; Silver et al., 2016; 2017).”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, background): “While there has been significant progress in developing deep reinforcement learning algorithms that learn complex skills and scale to high-dimensional observation spaces, such as pixels [1], [2], [3], [4], learning behaviors that generalize to new tasks and objects remains an open problem.”
Learning to Drive in a Day (2018, background): “In recent years, reinforcement learning (RL) – a machine learning subfield focused on solving Markov Decision Problems (MDP) [3] where an agent learns to select actions in an environment in an attempt to maximise some reward function – has shown an ability to achieve super-human results at games suc”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “[8] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “he best human Go players. Shortly thereafter AlphaGo was surpassed by AlphaGo Zero, which used less knowledge [65], and then AlphaZero which also learned to play chess and shogi at super-human levels [66]. Many successes above were in applications of RL to perfect information games (e.g., Go, chess,”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “Self-play in adversarial groups of agents is common in multi-agent reinforcement learning [45, 47].”
Multi-task Deep Reinforcement Learning with PopArt (2018, methodology): “Deep RL agents have been applied to board games such as Go (Silver et al. 2016) and chess (Silver et al. 2017), continuous control (Lillicrap et al. 2016; Duan et al. 2016), classic video-games such as Atari (Mnih et al. 2015; Hessel et al. 2018; Gruslys et al. 2018; Schulman et al. 2015; Schulman…”