Mastering Atari, Go, chess and shogi by planning with a learned model

By Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel and 2 more. Nature, 2019

2,607 citations recorded by the source corpus.

Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess1 and Go2, where a perfect simulator is available. However, in real-world problems, the dynamics governing the environment are often complex and unknown. Here we present the MuZero algorithm, which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. The MuZero algorithm learns an iterable model that produces predictions relevant to planning: the action-selection policy, the value function and the reward. When evaluated on 57 different Atari games3—the canonical video game environment for testing artificial intelligence techniques, in which model-based planning approaches have historically struggled4—the MuZero algorithm achieved state-of-the-art performance. When evaluated on Go, chess and shogi—canonical environments for high-performance planning—the MuZero algorithm matched, without any knowledge of the game dynamics, the superhuman performance of the AlphaZero algorithm5 that was supplied with the rules of the game. A reinforcement-learning algorithm that combines a tree-based search with a learned model achieves superhuman performance in high-performance planning and visually complex domains, without any knowledge of their underlying dynamics.

Subfields: Artificial Intelligence, Reinforcement Learning

View Paper: https://arxiv.org/abs/1911.08265

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “A human— working with an internal cognitive model— could quickly compensate; model-free deep reinforcement learning systems often can't, instead frequently requiring 15 Hair-splitters might argue that there is some sort of internal model that is self-generated, pointing to internal states of the sys”
Adversarial Training for High-Stakes Reliability (2022, background): “Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1, 2, 3, 4], robotics [5, 6], and language modeling and text-based reasoning [7, 8, 9, 10, 11].”
Toward Causal Representation Learning (2021, background): “The machine learning community has produced impressive successes with machine learning applications to big data problems [54], [148], [171], [223], [232].”
Towards Causal Representation Learning (2021, background): “successes with machine learning applications to big data problems [148, 171, 223, 231, 53].”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “We do not recommend to use the non-causal partial model together with MCTS with chance nodes.”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, background): “One potential approach to enable a robot to perform these manipulation tasks is with deep reinforcement learning (RL), which has achieved many successes in recent years [4, 5, 6, 7, 8].”
Learned Belief Search: Efficiently Improving Policies in Partially Observable Settings (2021, neutral): “Most recently MuZero combines RL and MCTS with a learned implicit model of the environment (Schrittwieser et al., 2019).”
A Generalist Agent (2022, methodology): “This is also true of state-of-the-art RL methods applied to board games (Schrittwieser et al., 2020).”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, background): “, 2016), the latter class of methods build explicit predictive models of the world and use those models to plan through or collect fictitious rollouts for model-free methods to learn from (Sutton, 1990; Ha & Schmidhuber, 2018; Kaiser et al., 2019; Schrittwieser et al., 2019).”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “MuZero (Schrittwieser et al., 2019) learns task-speciﬁc reward and value models to solve challenging tasks but requires large amounts of experience.”
Inductive biases for deep learning of higher-level cognition (2020, background): “Many machine learning systems have achieved excellent accuracy across a variety of tasks (Deng et al., 2009; Mnih et al., 2013; Schrittwieser et al., 2019), yet the question of whether their reasoning or judgement is correct has come under question, and answers seem to be wildly inconsistent,…”
Boosting Search Engines with Interactive Agents (2021, positive): “We also build an RL search agent based on MuZero (Schrittwieser et al., 2020) and BERT (Devlin et al.”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “In this way planning can be performed by rolling out models and evaluating such rollouts, [31, 32].”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “[91] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.”
Mastering Atari with Discrete World Models (2020, methodology): “On the other hand, the recently proposed MuZero agent (Schrittwieser et al., 2019) shows that planning can achieve impressive performance on board games and deterministic Atari games given extensive engineering effort and a vast computational budget.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, positive): “In video compression, Mandhane et al. [70] leverage the Muzero [237], one of the alpha series algorithms, to solve the safe RL problem in video compression.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “2018b) and limits us from running slower than real-time to perform more computationally expensive approaches (e.g. some forms of model-based planning Doya et al. 2002; Levine et al. 2019; Schrittwieser et al. 2019).”
Transformers are Sample Efficient World Models (2022, methodology): “We also note that IRIS outperforms MuZero, although the latter was not designed for the sample-efﬁcient regime.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”