From Pixels to Torques: Policy Learning with Deep Dynamical Models

By Niklas Wahlstrom, Thomas Bo Schön, Marc Peter Deisenroth. 2015

192 citations recorded by the source corpus.

Data-efficient learning in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. In this paper, we consider one instance of this challenge, the pixels-totorques problem, where an agent must learn a closed-loop control policy from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model that uses deep autoencoders to learn a low-dimensional embedding of images jointly with a prediction model in this low-dimensional feature space. This joint learning ensures that not only static properties of the data are accounted for, but also dynamic properties. This is crucial for long-term predictions, which lie at the core of the adaptive model predictive control strategy that we use for closedloop control. Compared to state-of-the-art reinforcement learning methods, our approach learns quickly, scales to high-dimensional state spaces and facilitates fully autonomous learning from pixels to torques.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Continuous control with deep reinforcement learning (2015, methodology): “However, due to the high computational demand, PILCO is “impractical for high-dimensional problems” [22].”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “Hence, we have set up an end-to-end optimization process, where the agent is encouraged to learn a “fast” reinforcement learning algorithm.”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “Keywords: policy gradient, reinforcement learning, deep learning, gradient estimation, temporal difference learning”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, methodology): “Other methods build a latent state-space model that is sufficient to reconstruct the observation stream at pixel level [48, 49], or to predict its future latent states [13, 11], which facilitates more efficient planning but still focuses the majority of the model capacity on potentially irrelevant d”
Recurrent World Models Facilitate Policy Evolution (2018, background): “Recent work along these lines [99, 100] was able to train controllers using the bottleneck hidden layer of an autoencoder as low-dimensional feature vectors to control a pendulum from pixel inputs.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “A similar approach also has been studied in robotics (Wahlström et al., 2015).”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Most of the above work focuses on modelling music, speech or other low-dimensional data, whereas here we present stochastic sequence models on high-dimensional pixel-based observations; noteworthy exception are found in (Watter et al., 2015; Wahlstr¨om et al., 2015).”
Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images (2015, methodology): “[25], where autoencoders are used to extract a latent representation from images, on which a non-linear model of the (latent) forward dynamics is learned with MLPs.”
Recurrent Environment Simulators (2017, background): “Deep neural networks have recently enabled significant advances in simulating complex environments, allowing for models that consider high-dimensional visual inputs across a wide variety of domains (Wahlström et al., 2015; Watter et al., 2015; Sun et al., 2015; Patraucean et al., 2015).”
World Models (2018, background): “Recent work along these lines (Wahlstrm et al., 2014; 2015) was able to train controllers using the bottleneck hidden layer of an autoencoder as low-dimensional feature vectors to control a pendulum from pixel inputs.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Model-based deep RL algorithms have been proposed for learning models and policies directly from raw pixel inputs [93], [94].”
Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning (2017, neutral): “Several works have proposed to use deep neural network models for building predictive models of images [29], but these methods have either required extremely large datasets for training [29] or were applied to short-horizon control tasks [30].”
A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning (2017, background): “In similar spirit to this paper, [32] extracts a latent representation from a PCA representation of the frames where controls can be applied.”
Combining self-supervised learning and imitation for vision-based rope manipulation (2017, background): “Impressive results have been obtained on playing Atari games [10], a large suite of robotic tasks in simulation [11], [12], [13], [14], [15], [16] and real world robotic manipulation tasks involving rigid objects in controlled settings [17], [18].”
Learning to Poke by Poking: Experiential Learning of Intuitive Physics (2016, background): “Only very recently, (Fragkiadaki et al., 2015; Wahlström et al., 2015; Watter et al., 2015; Oh et al., 2015) addressed the problem of model based control from vision in synthetic domains of manipulating two degree of freedom robotic arm, inverted pendulum, billiards and Atari games.”
Learning Visual Predictive Models of Physics for Playing Billiards (2015, background): “More recently works such as (Wahlstr¨om et al., 2015; Watter et al., 2015) have shown successful results on relatively simple domains of manipulating a synthetic two degree of freedom robotic arm or controlling an inverted pendulum in simulation.”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “NNs for model learning offer the potential to scale to higher dimensional problems with impressive sample complexity [22, 23, 24, 25].”
A Brief Survey of Deep Reinforcement Learning (2017, positive): “Based on the deep dynamical model [131], where high-dimensional observations are embedded into a lower-dimensional space using autoencoders, several model-based DRL algorithms have been proposed for learning models and policies from pixel information [81, 137, 132].”
SE3-nets: Learning rigid body motion using deep neural networks (2016, background): “in robotics and reinforcement learning, by mapping raw pixel images to low-dimensional (latent) encodings on top of which standard optimal control methods are applied [13], [14].”