Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

By Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, Wojciech Zaremba. 2016

270 citations recorded by the source corpus.

Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “Such approaches are termed Sim2Real [Golemo, 2018], and have been successfully applied [Christiano et al., 2016, Matas et al., 2018] in many scenarios.”
Solving Rubik's Cube with a Robot Hand (2019, background): “Domain adaption methods [112, 38], progressive nets [92], and learning inverse dynamics models [18] were all proposed to help with sim to real transfer.”
RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, methodology): “resentations of real and synthetic data [16], using generative models to make synthetic images more realistic [17], fine-tuning the policies trained in simulation in the real world [18], learning inverse dynamics models [19], multiobjective optimization of task fitness and transferability to the rea”
Learning dexterous in-hand manipulation (2018, background): “Domain adaption methods [66, 21], progressive nets [53], and learning inverse dynamics models [10] were all proposed to help with sim to real transfer.”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “The problem of making use of partially misspeciﬁed models has attracted attention in robotics, when transferring policies from simulation to real environments [29–31].”
Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks (2016, background): “Consequently, there has been a renewed interest in training models in the synthetic domain and applying them in real–world settings [8, 47, 38, 42, 25, 32, 35, 37].”
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization (2017, positive): “[17] transfered policies from simulation to a real robot by training an inverse-dynamics model from real world data.”
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping (2018, background): “Several recent works have looked at adapting to physical discrepancies explicitly [38], [39], [40], and incorporating these ideas into grasping is an exciting avenue for future work.”
Learning Latent Plans from Play (2019, background): “Goal conditioned policies have been explored extensively in the literature for reinforcement learning [23, 44, 36, 52, 3, 32, 48, 8, 58], as well as for control via inverse models [1, 35, 9, 61].”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “The vision API trained in the desired scenes, e.g. an object detection or semantic segmentation system, can bridge the gap between simulated environment and real-world (Tobin et al., 2017; Rusu et al., 2016; Christiano et al., 2016).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “(c) Utilising the potentially unlimited amount of training data that can be amassed in robotic simulators, several methods aim to transfer knowledge from the simulator to the real world [18, 97, 124].”
Preparing for the Unknown: Learning a Universal Policy with Online System Identification (2017, background): “More recently, deep neural networks have been applied to learn both forward dynamics [26] and inverse dynamics [6] from the real-world data, which can potentially model more complicated dynamics.”
Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task (2017, background): “Another example is [1], where deep inverse models are used on the simulated control policy to decide on suitable actions to take in the real world for a back-and-forth swing of an arm using position control.”
Benchmark Environments for Multitask Learning in Continuous Domains (2017, background): “…(Reacher, Hopper, Humanoid, HalfCheetah) and transferring between the modiﬁed environments; adding motor noise to the same set of environments (Christiano et al., 2016); simulated grasping and stacking using a Jaco arm (Rusu et al., 2016); and several custom grasping and manipulation tasks to…”
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, background): “The agent explores dangerous actions in simulation and transfers this knowledge to the real world [8]. this would require advances in transfer learning and in simulation.”
OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning (2017, methodology): “To demonstrate the effectiveness of OptionGAN in these settings, we use expert demonstrations from environments with varying gravity conditions as seen in (Hen-derson et al. 2017; Christiano et al. 2016).”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “policies from simulation to the real world [199]–[202].”
Policy transfer via modularity (2017, methodology): “A similar approach has been used for reaching, leaning, balancing, and other tasks that do not involve object interaction [25], [26], [27], [28].”