One-shot learning of manipulation skills with online dynamics adaptation and neural network priors

By Justin Fu, Sergey Levine, Pieter Abbeel. 2015

170 citations recorded by the source corpus.

One of the key challenges in applying reinforcement learning to complex robotic control tasks is the need to gather large amounts of experience in order to find an effective policy for the task at hand. Model-based reinforcement learning can achieve good sample efficiency, but requires the ability to learn a model of the dynamics that is good enough to learn an effective policy. In this work, we develop a model-based reinforcement learning algorithm that combines prior knowledge from previous tasks with online adaptation of the dynamics model. These two ingredients enable highly sample-efficient learning even in regimes where estimating the true dynamics is very difficult, since the online model adaptation allows the method to locally compensate for unmodeled variation in the dynamics. We encode the prior experience into a neural network dynamics model, adapt it online by progressively refitting a local linear model of the dynamics, and use model predictive control to plan under these dynamics. Our experimental results show that this approach can be used to solve a variety of complex robotic manipulation tasks in just a single attempt, using prior data from other manipulation behaviors.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, positive): “More recently, Fu et al. (2015) propose a model-based approach on top of iLQG with unknown dynamics (Levine & Abbeel, 2014), which uses samples collected from previous tasks to build a neural network prior for the dynamics, and can perform one-shot learning on new, but related tasks thanks to…”
Continuous Deep Q-Learning with Model-based Acceleration (2016, background): “Extending the bene-ﬁts of time-varying linear models to less restrictive settings is a promising direction and build on prior work (Levine et al., 2015; Fu et al., 2015).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “• Auxiliary Tasks (Mirowski et al., 2016; Jaderberg et al., 2016; Levine et al., 2016; Yu et al., 2018; Riedmiller et al., 2018): Uses additional supervised or unsupervised tasks (e.g., regressing depth images from color images, detecting loop closures, predicting end-effector poses) alongside the…”
Model-Ensemble Trust-Region Policy Optimization (2018, background): “This relieves the neural network from memorizing the input state, especially when the change is small (Deisenroth & Rasmussen, 2011; Fu et al., 2016; Nagabandi et al., 2017).”
Model-Based Reinforcement Learning via Meta-Policy Optimization (2018, background): “Several works approach this problem of model-bias by learning a distribution of models [26, 7, 10, 23], or by learning adaptive models [27, 28, 29].”
Dynamics-Aware Unsupervised Discovery of Skills (2019, background): “More recently, due to the challenges of applying these methods to high-dimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi et al., 2018; Chua et al., 2018b; Gal et al., 2016; Fu et al., 2016; Lenz et al., 2015) to learn dynamics models.”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, background): “Work in model adaptation (Levine and Koltun, 2013; Gu et al., 2016; Fu et al., 2015; Weinstein and Botvinick, 2017) has shown that a perfect global model is not necessary, and prior knowledge can be ﬁne-tuned to handle small changes.”
Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model (2016, neutral): “An alternative is to learn dynamics models in an on-line fashion, constantly adapting the model based on an incoming stream of observed states and actions [11] [28] [43] [22].”
Learning Awareness Models (2018, methodology): “The work of Fu et al. (2016) also ﬁts dynamics models using neural networks and uses planning in these models to guide action selection.”
Deep reinforcement learning: a survey (2020, methodology): “The method which combines knowledge from previous tasks with online adaptation of the dynamics model (Fu et al., 2016) helps solve a variety of complex robotic manipulation tasks in a single attempt.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, methodology): “These restrictions are relaxed in Iterative LQR [117] and Differential Dynamic Programming [376, 429], in which nonlinear dynamics and nonquadratic costs can be approximated as locally linear and locally quadratic, respectively.”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, methodology): “This approach is often used when applying model predictive control (Feldbaum, 1960; C. Xie, Patil, Moldovan, Levine, & Abbeel, 2016; Fu, Levine, & Abbeel, 2016; Janner et al., 2018; Deisenroth & Rasmussen, 2011), also often called system identiﬁcation (Chiuso & Pillonetto, 2019).”
Pick and Place Without Geometric Object Models (2017, background): “Since this survey, deep RL has become prominent in robotic manipulation [4], [5], [21], [22], [23].”
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks (2018, background): “However, only few works focused on contact-rich tasks [23] and none relied on haptic feedback in simulation, most likely because of the lack of fidelity of contact simulation and collision modeling for articulated rigid-body systems [20, 24].”
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks (2019, background): “However, only few works focused on contact-rich tasks [32] and none relied on haptic feedback in simulation, most likely because of the lack of fidelity of contact simulation and collision modeling for articulated rigid-body systems [33], [34], which would make it difficult to transfer both the cont”
Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials (2020, methodology): “One can also simply adapt a neural network online at test-time via SGD without MAML for one/few-shot learning (Fu, Levine, and Abbeel 2016).”
Electronic skins and machine learning for intelligent soft robots (2020, background): “A general trend observed in such works is the success of model-based RL ( 67 ) or learning by demonstration ( 68 ), approaches that leverage techniques from control theory or human knowledge, respectively.”