Learning Latent Dynamics for Planning from Pixels

By Danijar Hafner, Timothy P. Lillicrap, Ian S. Fischer, Ruben Villegas, David R Ha, Honglak Lee, James Davidson. 2018

1,944 citations recorded by the source corpus.

Planning has been very successful for control tasks with known environment dynamics. To leverage planning in unknown environments, the agent needs to learn the dynamics from interactions with the world. However, learning dynamics models that are accurate enough for planning has been a long-standing challenge, especially in image-based domains. We propose the Deep Planning Network (PlaNet), a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space. To achieve high performance, the dynamics model must accurately predict the rewards ahead for multiple time steps. We approach this using a latent dynamics model with both deterministic and stochastic transition components. Moreover, we propose a multi-step variational inference objective that we name latent overshooting. Using only pixel observations, our agent solves continuous control tasks with contact dynamics, partial observability, and sparse rewards, which exceed the difficulty of tasks that were previously solved by planning with learned models. PlaNet uses substantially fewer episodes and reaches final performance close to and sometimes higher than strong model-free algorithms.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Diffusion Autoencoders: Toward a Meaningful and Decodable Representation (2021, background): “Rich image representations are useful for many downstream tasks; for example, VAE are often used in model-based reinforcement learning [14, 17, 18] for predicting future outcomes of the environment.”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “It has been hypothesized that deep, stochastic models may mitigate the problems of compounding error [14, 20].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “DeepMDPs can be viewed as a formalization of recent works which use neural networks to learn latent space models of the environment (Ha & Schmidhuber, 2018; Oh et al., 2017; Hafner et al., 2018).”
Model-Based Reinforcement Learning for Atari (2019, neutral): “, 2018) and simulated (Watter et al., 2015; Hafner et al., 2018) robotic control.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Many such methods have been known to exhibit excellent performance in conventional off-policy settings, where additional data collection is allowed, but prior data is also utilized (Sutton, 1991; Watter et al., 2015; Zhang et al., 2018; Hafner et al., 2018; Janner et al., 2019).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “If the dynamics of the world are also learned, then it can be applied to planning [65, 39] or model-based RL [33, 50].”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “This family of models, sometimes called models with overshoot , can for instance be found in (Oh et al., 2017; Luo et al., 2019; Guo et al., 2018; Hafner et al., 2018; Gregor et al., 2019; Asadi et al., 2019) and MuZero (Schrittwieser et al., 2019) and is illustrated in Figure 3(d).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “Generative environment models trained with overshooting have been explored in the context of model-based planning [31, 32, 33, 34].”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “It is suﬃcient to act optimally given the dense rewards in our tasks, but could readily be extended to multi-step MPC, other existing planning algorithms as in Hafner et al. (2018) or Monte-Carlo Tree Search (Silver et al., 2016).”
IIRC: Incremental Implicitly-Refined Classification (2020, background): “Deep learning algorithms have led to transformational breakthroughs in computer vision [12, 17], natural language processing [19, 50], speech processing [3, 5], reinforcement learning [36, 44], robotics [16, 1], recommender”
SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation (2020, methodology): “Since PlaNet [54] learns an autoencoder and a dynamics model, we can visualize the future predicted observations of a trained PlaNet model.”
Reinforcement Learning with Augmented Data (2020, methodology): “Dreamer [27] and PlaNet [26] learn world models and use them to generate synthetic rollouts similar to Dyna [32].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, methodology): “The PlaNet benchmark [22] consists of six challenging control tasks from [43] with different traits.”
Challenges of Real-World Reinforcement Learning (2019, methodology): “Recent Model-based deep RL approaches (Hafner et al., 2018; Chua et al., 2018), where the algorithm plans against a learned transition model of the environment, show a lot of promise for improving sample efficiency.”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “…learns a latent space world model on top of VAE features Ha & Schmidhuber (2018) and builds value functions on top; (iii) PlaNet and (iv) Dreamer (Hafner et al., 2018; 2019) both of which learn a latent space world model and explicitly plan through it; (v) Pixel SAC: Vanilla SAC operating…”
Exploring Model-based Planning with Policy Networks (2019, methodology): “Latent dynamics models using VAE [19] are commonly used to solve problems with image input [12, 13, 15, 18].”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “(10) We implement the transition model as a recurrent state space model (RSSM; Hafner et al., 2018), the representation model by combining the RSSM with a convolutional neural network (CNN; LeCun et al.”
PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Learning to Plan from Pixels: Another related work is learning dynamics models for model-based RL [6]–[8].”
Block Contextual MDPs for Continual Learning (2021, background): “Our work is also related to the general problem of training a policy on Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998; Igl et al., 2018; Zhang et al., 2019; Han et al., 2019; Hafner et al., 2019) that capture both nonstationarity and rich observation settings.”