On Pre-Trained Image Features and Synthetic Images for Deep Learning

By Stefan Hinterstoißer, Vincent Lepetit, Paul Wohlhart, Kurt Konolige. 2017

246 citations recorded by the source corpus.

Deep Learning methods usually require huge amounts of training data to perform at their full potential, and often require expensive manual labeling. Using synthetic images is therefore very attractive to train object detectors, as the labeling comes for free, and several approaches have been proposed to combine synthetic and real images for training. In this paper, we show that a simple trick is sufficient to train very effectively modern object detectors with synthetic images only: We freeze the layers responsible for feature extraction to generic layers pre-trained on real images, and train only the remaining layers with plain OpenGL rendering. Our experiments with very recent deep architectures for object recognition (Faster-RCNN, R-FCN, Mask-RCNN) and image feature extractors (InceptionResnet and Resnet) show this simple approach performs surprisingly well.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Driving Policy Transfer via Modularity and Abstraction (2018, background): “sfer from simulation to reality remains difﬁcult even given highﬁdelity simulation [30,38,47,54], although successful examples exist for tasks such as optical ﬂow estimation [11] and object detection [22,19]. Work on transfer of sensorimotor control policies has mainly dealt with manual grasping and”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, background): “Intuitively, fine-tuning [18,42] is able to achieve significant performance in fully supervised learning tasks, but is challenged by transfer learning.”
DeceptionNet: Network-Driven Domain Randomization (2019, methodology): “Recent methods [15, 25, 10, 48] prerender a number of synthetic images featuring different light conditions.”
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization (2018, methodology): “[11] used synthetic data generated by adding Gaussian noise to the object of interest and Gaussian blurring the object edges before composing over a background image.”
Implicit 3D Orientation Learning for 6D Object Detection from RGB Images (2018, background): “[13] showed that by training only the head network of FasterRCNN [29] with randomized synthetic views of a textured 3D model, it also generalizes well to real images.”
Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data (2018, background): “[21] eschewed the idea of photorealism and generated images by adding Gaussian noise to the foreground of the rendered image and performing Gaussian blurring on the object edges to better integrate with the background image.”
Synthetic Data for Deep Learning (2019, result): “etic data; this runs contrary to the conclusions of the works [601,642] that we discuss below. An interesting method of using synthetic data for object detection was proposed by Hinterstoisser et al. [246]. They note that training on purely synthetic data may give sub-par results due to the low-leve”
DPOD: 6D Pose Object Detector and Refiner (2019, methodology): “Similar to [22, 15, 13] we exploit an idea of using a network already pre-trained on ImageNet as a backbone architecture.”
Segment Anything in High Quality (2023, background): “Then, some prompt-based learning works [16, 27, 17] are proposed to help these pre-trained models generalize to the downstream tasks instead of fine-tuning the internal model parameters [15] for better transfer learning.”
BOP Challenge 2020 on 6D Object Localization (2020, methodology): “In contrast, DNN-based methods require large amounts of annotated training images, which have been typically obtained by OpenGL rendering of the 3D object models on top of random backgrounds [31,43,19,11].”
Deep Template-based Object Instance Detection (2021, background): “The difficulty of gathering a real dataset can be partially alleviated using simple rendering techniques [14, 22, 30] combined with data augmentation such as random backgrounds and domain randomization [36, 43, 37], but still suffers from a domain gap with real images.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “In domains with many objects where labeling is impractical, scanning and simulating objects may not be [25, 26].”
ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape (2018, background): “Since annotating monocular data with metrically accurate 3D annotations is usually costly and difficult, many recent works leverage synthetic data [10, 8, 2, 1] to train their methods [20, 17, 35].”