A self-supervised learning system for object detection using physics simulation and multi-view pose estimation

By Chaitanya Mitash, Kostas E. Bekris, Abdeslam Boularias. 2017

118 citations recorded by the source corpus.

Progress has been achieved recently in object detection given advancements in deep learning. Nevertheless, such tools typically require a large amount of training data and significant manual effort to label objects. This limits their applicability in robotics, where solutions must scale to a large number of objects and variety of conditions. This work proposes an autonomous process for training a Convolutional Neural Network (CNN) for object detection and pose estimation in robotic setups. The focus is on detecting objects placed in cluttered, tight environments, such as a shelf with multiple objects. In particular, given access to 3D object models, several aspects of the environment are physically simulated. The models are placed in physically realistic poses with respect to their environment to generate a labeled synthetic dataset. To further improve object detection, the network self-trains over real images that are labeled using a robust multi-view pose estimation process. The proposed training process is evaluated on several existing datasets and on a dataset collected for this paper with a Motoman robotic arm. Results show that the proposed approach outperforms popular training processes relying on synthetic — but not physically realistic — data and manual annotation. The key contributions are the incorporation of physical reasoning in the synthetic data generation process and the automation of the annotation process over real images.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Representation & Self-Supervised Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “…of the preceding approaches require reward functions or labeled data, which can be difﬁcult to obtain in the real world, Mitash and collaborators [26] explore pretraining an object detector using realistic rendered images with randomized lighting from 3D models to bootstrap an automated learning…”
Learning to Play with Intrinsically-Motivated Self-Aware Agents (2018, background): “, 2017), pose estimation (Mitash et al., 2017), solving jigsaw puzzles (Noroozi & Favaro, 2016), colorization (Zhang et al.”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, background): “More recently, deep learning based approaches in computer vision are being adopted for the task of pose estimation of specific objects[33, 53, 54].”
Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects (2018, background): “For alternate approaches to related problems, see [33, 34, 35, 36, 37].”
On Pre-Trained Image Features and Synthetic Images for Deep Learning (2017, background): “complex scene composition [8,1,7,27] to achieve good results, and [15] stresses the need for photo-realistic rendering. Some approaches even use physics engines to enable realistic placing of objects [14]. This requires signiﬁcant resources and highly elaborate pipelines that are difﬁcult to enginee”
Self-supervised 6D Object Pose Estimation for Robot Manipulation (2019, methodology): “Previous works on self-supervised learning in robot perception mainly focus on learning object segmentation [6], object detection [7], and dense pixel-wise correspondences [29], [30], since it is relatively easy to obtain annotations in 2D image space.”
Deep Template-based Object Instance Detection (2021, methodology): “Our fully-annotated training dataset is generated with a physic-based simulator similar to [28], for which”
Implicit 3D Orientation Learning for 6D Object Detection from RGB Images (2018, background): “Photo-Realistic Rendering of object views and backgrounds has shown mixed generalization performance for tasks like object detection and viewpoint estimation [26,34,25,30].”