Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

By Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, Martin A. Riedmiller. 2015

915 citations recorded by the source corpus.

We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “…results, including playing Atari games from raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015), and acquiring advanced manipulation and locomotion skills (Levine et al., 2016; Lillicrap et al., 2015; Watter et al., 2015; Heess et al., 2015; Schulman et al., 2015; 2016).”
Density estimation using Real NVP (2016, background): “For example, in reinforcement learning, these invertible functions can help extend the set of functions for which an argmax operation is tractable for continuous Qlearning [23] or find representation where local linear Gaussian approximations are more appropriate [67].”
Benchmarking Deep Reinforcement Learning for Continuous Control (2016, background): “Notable examples are training agents to play Atari games based on raw pixels (Guo et al., 2014; Mnih et al., 2015; Schulman et al., 2015a) and to acquire advanced manipulation skills using raw sensory inputs (Levine et al., 2015; Lillicrap et al., 2015; Watter et al., 2015).”
End-to-End Training of Deep Visuomotor Policies (2015, methodology): “The training set for this supervised learning procedure can be constructed from example demonstrations [20], trajectory optimization under known dynamics [21, 22, 28], and trajectory-centric reinforcement learning methods that operate under unknown dynamics [19, 23], which is the approach taken in…”
Model-Based Reinforcement Learning for Atari (2019, background): “…several recent works have incorporated images into real-world (Finn et al., 2016; Finn & Levine, 2017; Babaeizadeh et al., 2017a; Ebert et al., 2017; Piergiovanni et al., 2018; Paxton et al., 2019; Rybkin et al., 2018; Ebert et al., 2018) and simulated (Watter et al., 2015; Hafner et al., 2019) .”
Learning Latent Dynamics for Planning from Pixels (2018, background): “E2C (Watter et al., 2015) and RCE (Banijamali et al., 2017) embed images into a latent space, where they learn local-linear latent transitions and plan for actions using LQR.”
Recurrent World Models Facilitate Policy Evolution (2018, background): “Learning a model of the dynamics from a compressed latent space enable RL algorithms to be much more data-efﬁcient [15, 101].”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “Directly learning models from pixels for RL is still an under-explored topic due to high demands on model accuracy and computational budget, but see (Finn & Levine, 2017; Watter et al., 2015; Wahlstr ¨ om et al., 2015).”
Value Iteration Networks (2016, methodology): “In future work we intend to learn different planning computations, based on simulation [10], or optimal linear control [30], and combine them with reactive policies, to potentially develop RL solutions for task and motion planning [14].”
Dueling Network Architectures for Deep Reinforcement Learning (2015, background): “Notable examples include deep Q-learning (Mnih et al., 2015), deep visuomotor policies (Levine et al., 2015), attention with recurrent networks (Ba et al., 2015), and model predictive control with embeddings (Watter et al., 2015).”
Continuous Deep Q-Learning with Model-based Acceleration (2016, methodology): “As in standard RL, model-based deep reinforcement learning methods have generally been more efﬁcient (Li & Todorov, 2004; Watter et al., 2015; Li & Todorov, 2004; Wahlstr ¨ om et al., 2015; Levine & Koltun, 2013), while model-free algorithms tend to be more generally applicable but substantially…”
Recurrent Environment Simulators (2017, background): “Deep neural networks have recently enabled significant advances in simulating complex environments, allowing for models that consider high-dimensional visual inputs across a wide variety of domains (Wahlström et al., 2015; Watter et al., 2015; Sun et al., 2015; Patraucean et al., 2015).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Many such methods have been known to exhibit excellent performance in conventional off-policy settings, where additional data collection is allowed, but prior data is also utilized (Sutton, 1991; Watter et al., 2015; Zhang et al., 2018; Hafner et al., 2018; Janner et al., 2019).”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, methodology): “This characteristic bears some similarity with approaches such as Embed to Control [12], which learns states that follow a linear dynamic, or the approach of [11], [13], [14] which learn states that make possible to reconstruct the next observation with models such as PSRs (predictive state…”
Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data (2016, methodology): “M L ] 2 0 M ay images, audio or other sensory inputs and (iii) scales to large data due to optimization of parameters based on stochastic gradient descent [2].”
Deep Kalman Filters (2015, methodology): “Recently, Watter et al. (2015) use temporal generative models for optimal control.”
World Models (2018, methodology): “Learning a model of the dynamics from a compressed latent space enable RL algorithms to be much more dataefficient (Finn et al., 2015; Watter et al., 2015; Finn, 2017).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Model-based deep RL algorithms have been proposed for learning models and policies directly from raw pixel inputs [93], [94].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, neutral): “Several works have proposed to use deep neural network models for building predictive models of images [29], but these methods have either required extremely large datasets for training [29] or were applied to short-horizon control tasks [30].”