Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models

By Yannis Assael, Niklas Wahlstrom, Thomas Bo Schön, Marc Peter Deisenroth. 2015

39 citations recorded by the source corpus.

Data-efficient reinforcement learning (RL) in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. We consider a particularly important instance of this challenge, the pixels-to-torques problem, where an RL agent learns a closed-loop control policy ("torques") from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model for learning a low-dimensional feature embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning is crucial for long-term predictions, which lie at the core of the adaptive nonlinear model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art RL methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces, is lightweight and an important step toward fully autonomous end-to-end learning from pixels to torques.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

Citations in the corpus

Towards Deep Symbolic Reinforcement Learning (2016, background): “Each of these shortcomings is an active area of research in the DRL community, where data efﬁcient learning [8, 9], transfer learning [10, 11, 12, 13], planning [14, 15], and transparency [16] are all hot topics.”
Learning to Learn for Global Optimization of Black Box Functions (2016, methodology): “This perspective also allows us to draw a connection between meta learning and model based active learning, where a model of the environment is used to guide an active process for collecting data to improve that model [Assael et al., 2015, Agrawal et al., 2016].”
Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networks (2016, background): “…instrumental in solving a number of challenging reinforcement learning (RL) problems, including high-dimensional robot control (Levine et al., 2015; Assael et al., 2015; Wat-ter et al., 2015), visual attention (Ba et al., 2015), and the Atari learning environment (ALE) (Guo et al., 2014; Mnih et…”
Learning to Perform Physics Experiments via Deep Reinforcement Learning (2016, positive): “Introducing actions into the prediction process amounts to learning a latent forward dynamics model, which can be exploited to plan actions to achieve novel goals (Watter et al., 2015; Assael et al., 2015; Fragkiadaki et al., 2016).”
SOLAR: Deep Structured Latent Representations for Model-Based Reinforcement Learning (2018, background): “for dynamical systems is typically aimed at improving reconstruction or forward prediction [18, 2, 41, 43], which largely benefits model-free RL and global model methods.”
Learning Actionable Representations with Goal-Conditioned Policies (2018, methodology): “In the context of RL and control, representation learning methods have been used for many down-stream applications (Lesort et al., 2018), including representing value functions (Barreto et al., 2016) and building models (Watter et al., 2015; Assael et al., 2015; Zhang et al., 2018b).”
Interactive Perception: Leveraging Action in Perception and Perception in Action (2016, methodology): “5 [Left] visualizes an example situation in which the robot has to estimate the potential articulation mechanism between two Lego blocks given only visual observations of a static scene.”
Generative Temporal Models with Memory (2017, background): “…also important components of intelligent agents, as they permit counterfactual reasoning, physical predictions, robot localisation, and simulation-based planning among other capacities (Sutton, 1991; Deisenroth and Rasmussen, 2011; Watter et al., 2015; Levine and Abbeel, 2014; Assael et al., 2015).”
Stable reinforcement learning with autoencoders for tactile and visual data (2016, methodology): “Recently, this approach was used to find optimal policies when only simulated images of the system are given [23], [24].”