Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience

By Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan D. Ratliff, Dieter Fox. 2018

623 citations recorded by the source corpus.

We consider the problem of transferring policies to the real world by training on a distribution of simulated scenarios. Rather than manually tuning the randomization of simulations, we adapt the simulation parameter distribution using a few real world roll-outs interleaved with policy training. In doing so, we are able to change the distribution of simulations to improve the policy transfer by matching the policy behavior in simulation and the real world. We show that policies trained with our method are able to reliably transfer to different robots in two real world tasks: swing-peg-in-hole and opening a cabinet drawer. The video of our experiments can be found at https://sites.google.com/view/simopt.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Robotics & Control

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “For example, in robotics and autonomous driving, a rapidly growing research topic is the study of simulation to real-world transfer: training policies with reinforcement learning in simulation and then transferring these policies into the real world (Sadeghi and Levine, 2017; Tan et al., 2018; Chebo”
Solving Rubik's Cube with a Robot Hand (2019, methodology): “[14] used real-world trajectories and a discrepancy metric to guide the distribution search.”
Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks (2018, background): “[9] have specifically looked into learning, from few realworld trajectories, the optimal distribution of such simulation properties, for transfer of policies learned in simulation to the real world.”
Sim2Real Transfer for Deep Reinforcement Learning with Stochastic State Transition Delays (2020, background): “Chebotar et al. [32] adapt the simulation parameters using real world roll-outs that can be used to match a ﬁxed delay between the simulator and real robot.”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “The closest related work to ours is [20], which ﬁnds the simulation parameters that bring the state distribution in simulation close to the state distribution in the real world.”
Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture (2021, background): “SimOpt [3] learns the randomization of simulation parameters based on a few real-world rollouts.”
Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (2024, background): “Open-world manipulation will require several advances, including scaling collections of simulated assets and tasks; few-shot sim-to-real (131); multi-task learning (114, 125); learning autonomously in the real world (120, 117, 54); learning reward functions from examples (120) or human videos…”
An Introduction to Deep Reinforcement Learning (2018, neutral): “manually tuning the randomization of simulations, one can also adapt the simulation parameters by matching the policy behavior in simulation to the real world (Chebotar et al., 2018).”
Synthetic Data for Deep Learning (2019, background): “In [97], the parameters of pφ(ξ) are learned automatically via a feedback loop from the results of real observations.”
On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward (2020, background): “Recent efforts aimed at simplifying calibration rely on machine learning and other statistical approaches to automate a labor-and time-intensive process (6, 42, 43).”
Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation (2020, background): “[8] learn to optimize simulator parameters for robotic control tasks by directly comparing trajectories between a real and a simulated robot.”
Learning to Walk in the Real World with Minimal Human Effort (2020, background): “Although researchers have developed principled approaches for mitigating the sim-to-real issue, including system identification [16], domain randomization [17, 18], and meta-learning [19, 20], it remains an open problem.”
RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation (2025, positive): “…camera pose, mass, friction and control latency combined with noise injection in expert demonstrations, enables sim-to-real transfer and robust visuomotor policies [35, 32, 5, 7, 23], and optimizing over worst-case ensembles further improves resilience to extreme domain shifts [33, 20, 22].”
Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? (2019, methodology): “Chebotar et al. [28] improve the simulation using the difference between simulation and reality observations.”
RoboTHOR: An Open Simulation-to-Real Embodied AI Platform (2020, background): “These largely address tasks such as object manipulation using robotic arms [11] and autonomous vehicles [58, 55].”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, background): “Some approaches to real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from expert demonstrations [27, 44, 45, 46, 47, 48].”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, positive): “This technique has proven successful in applications such as locomotion [26, 36], manipulation with complex dynamics [9, 75], and dexterous in-hand manipulation [2, 56, 72].”