How to Close Sim-Real Gap? Transfer with Segmentation!

By Mengyuan Yan, Qingyun Sun, Iuri Frosio, Stephen Tyree, Jan Kautz. 2020

10 citations recorded by the source corpus.

One fundamental difficulty in robotic learning is the sim-real gap problem. In this work, we propose to use segmentation as the interface between perception and control, as a domain-invariant state representation. We identify two sources of sim-real gap, one is dynamics sim-real gap, the other is visual sim-real gap. To close dynamics sim-real gap, we propose to use closed-loop control. For complex task with segmentation mask input, we further propose to learn a closed-loop model-free control policy with deep neural network using imitation learning. To close visual sim-real gap, we propose to learn a perception model in real environment using simulated target plus real background image, without using any real world supervision. We demonstrate this methodology in eye-in-hand grasping task. We train a closed-loop control policy model that taking the segmentation as input using simulation. We show that this control policy is able to transfer from simulation to real environment. The closed-loop control policy is not only robust with respect to discrepancies between the dynamic model of the simulated and real robot, but also is able to generalize to unseen scenarios where the target is moving and even learns to recover from failures. We train the perception segmentation model using training data generated by composing real background images with simulated images of the target. Combining the control policy learned from simulation with the perception model, we achieve an impressive $\bf{88\%}$ success rate in grasping a tiny sphere with a real robot.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

On the Transfer of Disentangled Representations in Realistic Settings (2020, background): “Moreover, sim2real transfer is of major interest in the robotic learning community, because of limited data and supervision in the real world (Tobin et al., 2017; Rusu et al., 2017; Peng et al., 2018; James et al., 2019; Yan et al., 2020; Andrychowicz et al., 2020).”
iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “These modalities can support research topics such as: depth/segmentation/normal/affordance prediction [48, 49, 50], action-conditioned flow prediction [51], multi-modal pose estimation [52, 53, 54], and visuomotor policy training assuming perfect vision systems [35, 55].”
iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes (2020, background): “as: depth/segmentation/normal/affordance prediction [45, 46, 47], action-conditioned flow prediction [48], multi-modal pose estimation [49, 50, 51], and visuomotor policy training assuming perfect vision systems [35, 52].”