Mastering Diverse Domains through World Models

By Danijar Hafner, J. Pašukonis, Jimmy Ba, Timothy P. Lillicrap. 2023

1,259 citations recorded by the source corpus.

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily applied to tasks similar to what they have been developed for, configuring them for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula. This achievement has been posed as a significant challenge in artificial intelligence that requires exploring farsighted strategies from pixels and sparse rewards in an open world. Our work allows solving challenging control problems without extensive experimentation, making reinforcement learning broadly applicable.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Harms from Increasingly Agentic Algorithmic Systems (2023, background): “For instance, DreamerV3 [87] collected diamonds from scratch without human data or curricula in MineCraft, which has been a longstanding challenge because the task is extremely complex and open-ended.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, background): “Video prediction can be regarded as a special form of generation, leveraging past observations to anticipate future frames [1, 9, 22, 26, 41, 59, 60, 65].”
TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “…between tasks, TD-MPC 2 formulates reward and value prediction as a discrete regression (multi-class classification) problem in a log -transformed space, which is optimized by minimizing cross-entropy with r t , q t as soft targets (Bellemare et al., 2017; Kumar et al., 2023; Hafner et al., 2023).”
Learning Interactive Real-World Simulators (2023, background): “With large transformer architectures, learning image-based world models has become plausible [58, 59, 60, 61, 62, 63], but mostly in games or simulated domains with visually simplistic and abundant data.”
Voyager: An Open-Ended Embodied Agent with Large Language Models (2023, background): “DreamerV3 [70], on the other hand, learns a world model to explore the environment and collect diamonds.”
GAIA-1: A Generative World Model for Autonomous Driving (2023, methodology): “World models have been successfully applied to control tasks in both simulation [9, 10, 11, 12, 13] and to real-world robotics tasks [14, 15].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Such properties can be useful for representation learning [35, 45, 88, 105, 131], model-based reinforcement learning [39, 40, 42, 43, 94, 96, 98], as well as model-predictive control [27, 30, 35, 41, 92, 104, 142].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “DreamerV3 [72] extends its focus to a broader range of domains with fixed hyperparameters.”
Locality Sensitive Sparse Encoding for Learning World Models Online (2024, background): “…the policy that maximizes the discounted cumulative reward: | ∼ Existing NN-based MBRL methods (Schrittwieser et al., 2020; Hansen et al., 2022; Hafner et al., 2023) have shown state-of-the-art performance on various domains but rely heavily on techniques to make data more stationary, such as…”
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (2023, methodology): “More recently, [15] utilized a learned world model to distill a policy that can efficiently explore in Minecraft.”
Do Embodied Agents Dream of Pixelated Sheep?: Embodied Decision Making using Language Guided World Modelling (2023, methodology): “Several existing agents overcome the problem of item scarcity in Minecraft by simplifying environment parameters such as action duration (Patil et al., 2020) or block break time (Hafner et al., 2023), making comparison between methods difficult.”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “…investigated world models in simulated tasks (Fragkiadaki et al., 2015; Ha and Schmidhuber, 2018; Hafner et al., 2019b,a; Hansen et al., 2022, 2023; Hafner et al., 2023; Schrittwieser et al., 2020; Samsami et al., 2024), as well as real-world locomotion and manipulation tasks (Lee et al., 2020;…”
Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory (2023, methodology): “For RL-based methods like VPT [2] and DreamerV3 [7], the goal item(diamond) is hard-coded into the model weights, which means there are no easy way to re-task the trained RL agents for collecting other items in the inference stage.”
Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models (2023, background): “Note that though tree-based search necessitates an environment model that can expand from an arbitrary state (Vodopivec et al., 2017), often requiring extra training in RL (Hafner et al., 2023), such a problem does not exist for most LM tasks.”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “For instance, [21, 37, 63, 60, 53, 22] train action-conditioned dynamics models and run RL in the model.”