Sim-to-Real Transfer of Robotic Control with Dynamics Randomization
By Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, Pieter Abbeel. 2017
1,765 citations recorded by the source corpus.
Simulations are attractive environments for training agents as they provide an abundant source of data and alleviate certain safety concerns during the training process. But the behaviours developed by agents in simulation are often specific to the characteristics of the simulator. Due to modeling error, strategies that are successful in simulation may not transfer to their real world counterparts. In this paper, we demonstrate a simple method to bridge this “reality gap”. By randomizing the dynamics of the simulator during training, we are able to develop policies that are capable of adapting to very different dynamics, including ones that differ significantly from the dynamics on which the policies were trained. This adaptivity enables the policies to generalize to the dynamics of the real world without any training on the physical system. Our approach is demonstrated on an object pushing task using a robotic arm. Despite being trained exclusively in simulation, our policies are able to maintain a similar level of performance when deployed on a real robot, reliably moving an object to a desired location from random initial configurations. We explore the impact of various design decisions and show that the resulting policies are robust to significant calibration error.
Subfields: Artificial Intelligence, Control and Systems Engineering, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Yoshua Bengio
- Mohit Bansal
- Ruslan Salakhutdinov
- Trevor Darrell
- Quoc V. Le
In Control and Systems Engineering
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Vikash Kumar
- Abhishek Gupta
- Abhinav Kumar Gupta
- Dieter Fox
- Li Fei-Fei
In Robotics & Control
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Abhinav Kumar Gupta
- Vikash Kumar
- Dieter Fox
- Li Fei-Fei
- Silvio Savarese
Citations in the corpus
- On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Moreover, sim2real transfer is of major interest in the robotic learning community, because of limited data and supervision in the real world (Tobin et al., 2017; Rusu et al., 2017; Peng et al., 2018; James et al., 2019; Yan et al., 2020; Andrychowicz et al., 2020).”
- Solving Rubik's Cube with a Robot Hand (2019, methodology): “This idea has also been extended to dynamics randomization [80, 3, 105, 119, 77] to learn a robust policy that transfers to similar environments but with different dynamics.”
- Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Authors in [118], randomized the dynamics of the simulator during training.”
- A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “Similarly, randomizing the dynamics of the simulator during training has also been proposed (Peng et al., 2017) to bridge the reality gap .”
- A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “However, Sim2Real [27]–[29] and robotics in the physical world will not be covered in this paper.”
- RoboNet: Large-Scale Multi-Robot Learning (2019, background): “A number of works have studied learning representations and policies that transfer across domains, including transfer from simulation to the real world [1, 33, 34], transfer across different dynamics [35, 36, 37, 2], transfer across robot morphologies with invariant feature spaces [38] and modularit”
- Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, neutral): “[22] use randomization of physical parameters of the scene to learn a policy in simulation and transfer it to a real robot for pushing a puck to a target position.”
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, methodology): “A robust controller can be learned by injecting noise to the simulation [19], adding perturbations to the simulated robot [37], leveraging multiple simulators [3], using domain randomization [48] and dynamics randomization [29, 40, 34].”
- Learning quadrupedal locomotion over challenging terrain (2020, positive): “Another is randomization of physical parameters that vary between simulation and reality, such that the controller becomes robust to a range of conditions that cover those that arise in physical deployment, without the necessity to precisely model these conditions a priori [21].”
- RLBench: The Robot Learning Benchmark & Learning Environment (2019, background): “…on-the-fly, meaning that you cam generate an infinite amount for your imitation learning algorithm. c) Sim-to-Real Transfer : Recently there has been a large amount of work in learning control policies in simulation and then transferring these to the real world [48], [49], [50], [51], [52], [53].”
- Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “While other works have shown that injecting noise into the dynamics either during imitation learning [33] or sim-to-real transfer [34] can alleviate cascading errors, we provide a simple method based only on data augmentation.”
- Learning dexterous in-hand manipulation (2018, methodology): “Following previous work on domain randomization [54, 62, 45], we randomize most of the aspects of the simulated environment in order to learn both a policy and a vision model that generalizes to reality.”
- Reinforcement and Imitation Learning for Diverse Visuomotor Skills (2018, background): “[32] address the transfer from simulation to reality, focusing on randomizing visual appearance and robot dynamics respectively.”
- Graph networks as learnable physics engines for inference and control (2018, methodology): “Our inference model shares similar aims as approaches for learning system identification explicitly (Yu et al., 2017; Peng et al., 2017), learning policies that are robust to hidden property variations (Rajeswaran et al., 2016), and learning exploration strategies in uncertain settings (Schmidhuber,…”
- Challenges of Real-World Reinforcement Learning (2019, background): “Domain randomization (Peng et al., 2018) involves explicitly training an agent on various perturbations of the environment and averaging these learning errors together during training.”
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Nominal dynamics models based on knowledge of physics [1], or through learning [2], complements a growing body of work on successful simulation to reality transfer and system identification [3, 4, 5, 6, 7].”
- Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias (2018, background): “However, beyond simple robotic picking tasks [1, 2, 3], there exist little support to this level of optimism.”
- Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “There have been more robotics works that do not use vision, but that apply domain randomization on physical properties of the simulator to aid transferability [40, 46, 1, 68, 43].”
- Understanding Continual Learning Settings with Data Distribution Drift Analysis (2021, background): “For example, in robotics, many approaches are focused on transferring a model trained in a simulated environment to real environments (Peng et al., 2018; Traor ´ e et al., 2019; James et al., 2019).”