Mastering Atari with Discrete World Models

By Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, Jimmy Ba. 2020

1,283 citations recorded by the source corpus.

Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, DreamerV2 reaches 200M frames and exceeds the final performance of the top single-GPU agents IQN and Rainbow.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “Nevertheless, even after extensive tuning of these objectives and attempts at similar objectives (e.g., we brieﬂy investigated incorporating ideas from Hafner et al. (2020)), we were not able to achieve any better results.”
Mastering Atari Games with Limited Data (2021, methodology): “Monte-Carlo Tree Search [1, 33, 34, 14], or MCTS, is a heuristic search algorithm.”
Temporal Difference Learning for Model Predictive Control (2022, positive): “TD-MPC consistently outperforms CURL and DrQ, and is competitive with DrQ-v2 and Dreamer-v2.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, background): “Additionally, world models can be a solution to the sample efficiency issues of RL algorithms by acting as a simulator of the environment [7, 85, 86, 62, 13, 15, 87], although this assumes the world model is an accurate model of the environment.”
Transformers are Sample Efficient World Models (2022, methodology): “Indeed, experience requirements range from months of gameplay for DreamerV2 (Hafner et al., 2021) in Atari 2600 games (Bellemare et al., 2013b) to thousands of years for OpenAI Five in Dota2 (Berner et al., 2019).”
Reasoning with Language Model is Planning with World Model (2023, background): “With latent imagination in a world model, RL agents can be trained to solve long-horizon tasks (Hafner et al., 2019, 2020).”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “However, learning high-quality world models compatible with existing end-to-end autonomous driving models is challenging, despite successful attempts in game simulations [19–21, 46 ] and laboratory robotics environments To address these challenges, we propose Drive-WM.”
Learning Interactive Real-World Simulators (2023, background): “With large transformer architectures, learning image-based world models has become plausible [58, 59, 60, 61, 62, 63], but mostly in games or simulated domains with visually simplistic and abundant data.”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “Recently, a model-based method, DreamerV2 [Hafner et al., 2020] was also shown to solve visual continuous control problems and it was ﬁrst to solve the humanoid locomotion problem from pixels.”
Genie: Generative Interactive Environments (2024, background): “…2018; Oh et al., 2015), which enable next-frame prediction that is conditioned on action inputs (Bamford and Lucas, 2020; Chiappa et al., 2017; Eslami et al., 2018; Hafner et al., 2020, 2021; Kim et al., 2020, 2021; Micheli et al., 2023; Nunes et al., 2020; Pan et al., 2022; Robine et al., 2023).”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “RL can be effectively combined with world models [82], [83], [84], though this presents speciﬁc challenges (See Section IV-C).”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Building upon this foundation, DreamerV2 [71] introduces a discrete latent state space along with an improved objective.”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “…of the RL. Recent works on representation learning focus on learning world models and use them to train the RL algorithm [Ha and Schmidhuber, 2018, Hafner et al., 2020] or learning state representations for improving the sample sample efﬁciency of RL algorithms using reconstruction-based…”
Mastering diverse control tasks through world models (2025, neutral): “The representations are sampled from a vector of softmax distributions and we take straight-through gradients through the sampling step 23 .”
DayDreamer: World Models for Physical Robot Learning (2022, methodology): “We leverage the Dreamer algorithm (Hafner et al., 2019; 2020) for online learning on physical robots, without the need for simulators.”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, background): “several latent categorical variables [25].”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “Similarly, the profiles of DreamerV2 and M-IQN for τ 8 intersect at multiple points.”