RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

By Daniel E. Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, Yunfei Bai. 2020

132 citations recorded by the source corpus.

The success of deep reinforcement learning (RL) and imitation learning (IL) in vision-based robotic manipulation typically hinges on the expense of large scale data collection. With simulation, data to train a policy can be collected efficiently at scale, but the visual gap between sim and real makes deployment in the real world difficult. We introduce RetinaGAN, a generative adversarial network (GAN) approach to adapt simulated images to realistic ones with object-detection consistency. RetinaGAN is trained in an unsupervised manner without task loss dependencies, and preserves general object structure and texture in adapted images. We evaluate our method on three real world tasks: grasping, pushing, and door opening. RetinaGAN improves upon the performance of prior sim-to-real methods for RL-based object instance grasping and continues to be effective even in the limited data regime. When applied to a pushing task in a similar visual domain, RetinaGAN demonstrates transfer with no additional real data requirements. We also show our method bridges the visual gap for a novel door opening task using imitation learning in a new visual domain. Visit the project website at retinagan.github.io

Subfields: Biophysics, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “To reduce the simulation-to-real gap we transform robot images via RetinaGAN [16] to look more realistic while preserving genera object structure.”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, methodology): “In order to reduce the simulation-to-real (sim-to-real) gap when deploying policies on real robots, we train a RetinaGAN [57] model, a version of CycleGAN [58], to transform simulated robot images to look more realistic while preserving general object structure, following the sim-to-real setup in [1”
Evaluating Real-World Robot Manipulation Policies in Simulation (2024, unknown): “Our goal is to construct a simulator S , for which there is a strong correlation between the relative performances in real and the relative performances in simulation R S ,a and R S ,b .”
Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items (2022, methodology): “Because these models are scanned rather than modeled by hand, they are realistic, not idealized, reducing the difficulty of transferring learning from simulation to real [65], [66], [67].”