Dream to Control: Learning Behaviors by Latent Imagination

By Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, Mohammad Norouzi. 2019

2,090 citations recorded by the source corpus.

To select effective actions in complex environments, intelligent agents need to generalize from past experience. World models can represent knowledge about the environment to facilitate such generalization. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks purely by latent imagination. We efficiently learn behaviors by backpropagating analytic gradients of learned state values through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Generative Models

Citations in the corpus

Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, neutral): “We compare the form of our representation learning objective to “latent space model” approaches based on bisimulation principles, popular in the RL literature [20, 41, 22, 13], and show that these objectives are, in contrast, very liable to aliasing issues even in simple scenarios, explaining their ”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “, 2020), with the rollout in latent space sometimes referred to as ‘imagination’ (Racanière et al., 2017; Hafner et al., 2019).”
Reinforcement Learning with Augmented Data (2020, methodology): “In recent times, these ideas have been used to improve the performance of systems that have explicitly learned world models of the environment and generated synthetic rollouts using them [33, 11, 27].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “We compare DrQ to Dreamer [21] and the upper-bound performance of SAC [20] from states 4 .”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “…learns a latent space world model on top of VAE features Ha & Schmidhuber (2018) and builds value functions on top; (iii) PlaNet and (iv) Dreamer (Hafner et al., 2018; 2019) both of which learn a latent space world model and explicitly plan through it; (v) Pixel SAC: Vanilla SAC operating…”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, methodology): “This has been demonstrated by learning dynamics using future predictions (Watter et al., 2015; Oh et al., 2017; Ebert et al., 2017; Banijamali et al., 2018; Ha & Schmidhuber; Hafner et al., 2018; Ichter & Pavone, 2019; Hafner et al., 2020; Zhang et al., 2019), learning belief representations (Gregor”
Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models (2023, methodology): “For instance, [21, 37, 63, 60, 53, 22] train action-conditioned dynamics models and run RL in the model.”
A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning (2022, background): “…reinforcement learning (MBRL) (Janner et al., 2019; Kaiser et al., 2019; Schrittwieser et al., 2020; Zhang et al., 2019b; van Hasselt et al., 2019; Hafner et al., 2019b;a; Lenz et al., 2015), which explicitly builds a predictive model to generate samples for learning RL policy, has been widely…”
Simple and Effective VAE Training with Calibrated Decoders (2020, positive): “human pose [38], as well as model-based reinforcement learning [21, 20, 19], and representation learning [31, 46, 41].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, positive): “Such properties can be useful for representation learning [35, 45, 88, 105, 131], model-based reinforcement learning [39, 40, 42, 43, 94, 96, 98], as well as model-predictive control [27, 30, 35, 41, 92, 104, 142].”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “The synergy between model-based RL [9], [10], [96] and video prediction is well defined as the latter aims to model an accurate representation of high-dimensional environments, while the former uses the learned world models as a context for decision-making.”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, methodology): “Alternatively, model-based approaches [26, 27], which explicitly learn the environment forward dynamics from images have also been employed for multi-task vision-based robotic manipulation tasks, either with planning algorithms [28, 29, 30, 31, 32, 33, 34, 35] or for optimizing a parametric policy…”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “Per common practice [Hafner et al., 2019], we employ action repeat of 2 and measure sample complexity in the environment steps, rather than the actor steps.”
DayDreamer: World Models for Physical Robot Learning (2022, methodology): “We leverage the Dreamer algorithm (Hafner et al., 2019; 2020) for online learning on physical robots, without the need for simulators.”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, methodology): “This model, known as a world model [19, 20, 35], is designed to predict future states based on current states and ego actions.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “autoregressive network used to predict future latents [6], [128], [133] similar to how world models are constructed in reinforcement learning [75], [76].”
Planning to Explore via Self-Supervised World Models (2020, methodology): “Baselines We compare our agent to a state-of-the-art task-oriented agent that receives rewards throughout training, Dreamer (Hafner et al., 2020).”
Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstruction (2020, methodology): “The main objective of this experiment is to demonstrate that Dreaming has advantages over the baseline method Dreamer [16] on difficult 5 manipulation tasks exhibited in Fig.”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “Such methods include contrastive learning with data augmentation [25, 26, 27], dynamics prediction [28, 29], bi-simulation [30], temporal or goal distance [31, 32], or domain specific information [33].”