Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation

By Kuan Fang, Yunfei Bai, Stefan Hinterstoißer, Mrinal Kalakrishnan. 2017

129 citations recorded by the source corpus.

Learning-based approaches to robotic manipulation are limited by the scalability of data collection and accessibility of labels. In this paper, we present a multi-task domain adaptation framework for instance grasping in cluttered scenes by utilizing simulated robot experiments. Our neural network takes monocular RGB images and the instance segmentation mask of a specified target object as inputs, and predicts the probability of successfully grasping the specified object for each candidate motor command. The proposed transfer learning framework trains a model for instance grasping in simulation and uses a domain-adversarial loss to transfer the trained model to real robots using indiscriminate grasping data, which is available both in simulation and the real world. We evaluate our model in real-world robot experiments, comparing it with alternative model architectures as well as an indiscriminate grasping baseline.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Li Fei-Fei
3. Roozbeh Mottaghi
4. Aniruddha Kembhavi
5. Chelsea Finn
6. Ali Farhadi
7. Silvio Savarese
8. Mohit Bansal

In Control and Systems Engineering

1. Sergey Levine
2. Yuke Zhu
3. Silvio Savarese
4. Li Fei-Fei
5. Chelsea Finn
6. Jie Tan
7. Animesh Garg
8. Ted Xiao

Citations in the corpus

• Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “feature level [49, 12] or pixel level [4] to transfer the controller.”
• RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer (2020, background): “Or, feature-level adaptation aligns intermediate network features between the domains [15], [16], [17].”
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping (2018, background): “obtain labels in simulation and use domain adaptation to generalize the policy to real-world scenes, which requires solving a simulation-to-reality transfer problem [19].”
• KETO: Learning Keypoint Representations for Tool Manipulation (2019, background): “When collecting robot data is expensive and time-consuming in the real world, such self-supervised learning can be conducted in simulated environments [7], [50], [15], [16].”
• PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, methodology): “Our image-based implementation is similar to Fang et al. [60], which utilizes image segmentation masks for task conditioning.”
• ManipulaTHOR: A Framework for Visual Object Manipulation (2021, background): “A long-standing problem in robotics research is object manipulation [12, 4, 5, 31, 39, 6, 9, 21, 10].”
• Learning task-oriented grasping for tool manipulation from simulated self-supervision (2018, background): “families depending on the source of supervision: 1) supervised learning approaches [30, 35, 46], where the models are trained with a dataset of objects with ground-truth grasp annotations, and 2) self-supervision approaches [23, 31, 42, 3, 11], where grasp labels are automatically generated by a rob”