A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play
By David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel and 3 more. 2018
4,131 citations recorded by the source corpus.
One program to rule them all Computers can beat humans at increasingly complex games, including chess and Go. However, these programs are typically constructed for a particular game, exploiting its properties, such as the symmetries of the board on which it is played. Silver et al. developed a program called AlphaZero, which taught itself to play Go, chess, and shogi (a Japanese version of chess) (see the Editorial, and the Perspective by Campbell). AlphaZero managed to beat state-of-the-art programs specializing in these three games. The ability of AlphaZero to adapt to various game rules is a notable step toward achieving a general game-playing system. Science, this issue p. 1140; see also pp. 1087 and 1118 AlphaZero teaches itself to play three different board games and beats state-of-the-art programs in each. The game of chess is the longest-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. By contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go by reinforcement learning from self-play. In this paper, we generalize this approach into a single AlphaZero algorithm that can achieve superhuman performance in many challenging games. Starting from random play and given no domain knowledge except the game rules, AlphaZero convincingly defeated a world champion program in the games of chess and shogi (Japanese chess), as well as Go.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Mohit Bansal
- Pieter Abbeel
- Quoc V. Le
- Ruslan Salakhutdinov
- Jianfeng Gao
- Percy Liang
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Yuxi Li
- Ofir Nachum
- Nicolas Manfred Otto Heess
- Timothy P. Lillicrap
Citations in the corpus
- Risks from Learned Optimization in Advanced Machine Learning Systems (2019, methodology): “For example, Go, Chess, and Shogi have a very large domain of possible board states, but admit a single high-level strategy for play— heuristic-guided tree search—that performs well across all board states.[10] On the other hand, a classifier trained on random noise is unlikely to benefit from compr”
- Measuring the Algorithmic Efficiency of Neural Networks (2020, background): “It seems to us as if language modeling [Devlin et al., 2018, Radford et al., , Raffel et al., 2019] and playing games [Silver et al., 2016, Silver et al., 2017, Silver et al., 2018,OpenAI et al., 2019] are the domains of interest given our criteria.”
- Dota 2 with Large Scale Deep Reinforcement Learning (2019, methodology): “For comparison Chess requires around one thousand values per observation (mostly 6-possibility categorical values) and Go around six thousand values (all binary)[12].”
- Structured agents for physical construction (2019, methodology): “The efficiency of Monte-Carlo Tree Search (MCTS) (Coulom, 2006) planning in RL has recently been highlighted in (Guo et al., 2014; Silver et al., 2016; 2017; 2018).”
- Rehearsal revealed: The limits and merits of revisiting samples in continual learning (2021, background): “Recent advances of neural networks have shown promising results by surpassing human capabilities in a wide range of tasks [39, 32, 38].”
- Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “Deep reinforcement learning (RL) has shown an immense potential to achieve superhuman performance (Mnih et al., 2013; Silver et al., 2018) on some narrow and well-defined”
- Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2016) and achieving super human performance (Silver et al., 2018), it is not immediately clear if the aforementioned benchmarks have the sufficient characteristics of desired environments for continual RL.”
- MOReL : Model-Based Offline Reinforcement Learning (2020, background): “Unfortunately, a direct embodiment of this trial and error learning is often inefficient and feasible only with a simulator [9, 10, 11].”
- Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “[55] D.”
- Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “Several recent model-based RL methods use latent state representations as a way to simplify the model learning and policy rollout elements of model-based policy optimization (Oh et al., 2017; Silver et al., 2018; Hafner et al., 2020), with the rollout in latent space sometimes referred to as ‘imagin”
- Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “This can be thought of as a scaled-down and simplified version of the AlphaZero algorithm (Silver et al., 2018), optionally coupled with an environment transition model, similar to the set-up in (van Hasselt et al.”
- Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings (2021, background): “Most of these successes were achieved in fully observable games such as Backgammon (Tesauro, 1994), Chess (Campbell et al., 2002) and Go (Silver et al., 2016; 2017; 2018).”
- Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization (2021, background): “Most recent successes in games are based on decentralized multi-agent learning (Brown, 1951; Singh et al., 2000; Lowe et al., 2017; Silver et al., 2018), where agents compete against each other and optimize their own rewards to gradually improve their strategies.”
- Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning (2019, methodology): “This includes examples such as Go (Silver et al., 2016; 2017; 2018), poker (Brown & Sandholm, 2017; Moravˇcík et al., 2017; Brown & Sandholm, 2019) and chess (Campbell et al., 2002).”
- The Hanabi Challenge: A New Frontier for AI Research (2019, background): “thereafter AlphaGo was surpassed by AlphaGo Zero, which used less knowledge [69], and then AlphaZero which also learned to play chess and shogi at super-human levels [70].”
- Continual learning: A comparative study on how to defy forgetting in classification tasks (2019, background): “In recent years, machine learning models have been reported to exhibit or even surpass human level performance on individual tasks, such as Atari games [1] or object recognition [2].”