Mastering Atari Games with Limited Data

By Weirui Ye, Shao-Wei Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao. 2021

335 citations recorded by the source corpus.

Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at https://github.com/YeWR/EfficientZero. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Adversarial Training for High-Stakes Reliability (2022, background): “Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1, 2, 3, 4], robotics [5, 6], and language modeling and text-based reasoning [7, 8, 9, 10, 11].”
A Cookbook of Self-Supervised Learning (2023, methodology): “The best-performing method doing this is EfficientZero [Ye et al., 2021], which modifies MuZero [Schrittwieser et al., 2020] by, among other modifications, adding the SimSiam [Chen and He, 2020] objective to train the encoder and the forward model, and sets the new state of the art on Atari100k.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, positive): “Monte Carlo Tree Search (MCTS) is a heuristic search al-gorithm that is proved successful on many decision-making environments, such as Atari (Ye et al., 2021) and Go (Silver et al., 2016).”
Transformers are Sample Efficient World Models (2022, methodology): “Recently, world models were leveraged in several ways: pure representation learning (Schwarzer et al., 2021), lookahead search (Schrittwieser et al., 2020; Ye et al., 2021), and learning in imagination (Ha & Schmidhuber, 2018; Kaiser et al., 2020; Hafner et al., 2020; 2021).”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “deep learning to RL systems produced superhuman performance on a wider variety of narrow tasks with limited to no human supervision, including but not limited to increasingly complex board games [35, 146, 168, 169] and video games [129, 161, 199].”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, background): “TD-MPC 2 – along with several other contemporary RL methods (Yarats et al., 2021; Ye et al., 2021; Hafner et al., 2023) – seek to democratize use of RL ( i.e. , lowering the barrier of entry for smaller teams of academics, practitioners, and individuals with fewer resources) by improving robustness…”
Mastering diverse control tasks through world models (2025, neutral): “EfficientZero 40 holds the state of the art by combining online tree search, prioritized replay and hyperparameter scheduling, but also resets levels early to increase data diversity, making a comparison difficult.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “They have proven to be highly effective in game environments or board games [9, 84].”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “We consider: (i) our method implemented using a state predictor ( h θ being the identity function), (ii) our method implemented without the latent consistency loss from Equation 10, and lastly: the consistency loss replaced by either (iii) the reconstruction objective of PlaNet and Dreamer, or (iv) ”