Learning Deep Object Detectors from 3D Models

By Xingchao Peng, Baochen Sun, Karim Ali, Kate Saenko. 2014

395 citations recorded by the source corpus.

Crowdsourced 3D CAD models are easily accessible online, and can potentially generate an infinite number of training images for almost any object category. We show that augmenting the training data of contemporary Deep Convolutional Neural Net (DCNN) models with such synthetic data can be effective, especially when real training data is limited or not well matched to the target domain. Most freely available CAD models capture 3D shape but are often missing other low level cues, such as realistic object texture, pose, or background. In a detailed analysis, we use synthetic CAD images to probe the ability of DCNN to learn without these cues, with surprising findings. In particular, we show that when the DCNN is fine-tuned on the target detection task, it exhibits a large degree of invariance to missing low-level cues, but, when pretrained on generic ImageNet classification, it learns better when the low-level cues are simulated. We show that our synthetic DCNN training approach significantly outperforms previous methods on the benchmark PASCAL VOC2007 dataset when learning in the few-shot scenario and improves performance in a domain shift scenario on the Office benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “In [31], the authors ﬁnd that by pretraining a network on ImageNet and ﬁne-tuning on synthetic data created from 3D models, better detection performance on the PASCAL dataset can be achieved than training with only a few labeled examples from the real dataset.”
Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “various prediction tasks, including gaze estimation [43], text detection and classification in RGB images [9, 15], font recognition [42], object detection [10, 27], hand pose estimation in depth images [38, 37], scene recognition in RGB-D [11], semantic segmentation of urban scenes [31], and human p”
A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective (2018, background): “A well known use case is the ImageNet image classiﬁ-cation dataset [104] where tens of millions of images were organized according to a semantic hierarchy by WordNet using Amazon Mechanical Turk.”
RenderGAN: Generating Realistic Labeled Data (2016, methodology): “Peng et al. (2015) generated training data for a DCNN with 3D-CAD models.”
Recognition in Terra Incognita (2018, methodology): “[48,49,50,51,52] look at methods for adapting detectors from one data source to another, such as from synthetic to real data or from images to video.”
Sim-to-Real Robot Learning from Pixels with Progressive Nets (2016, background): “om simulation to reality for domains using end-to-end (pixel-to-action) learning. A growing body of work has been investigating the ability of deep networks to transfer between domains. Some research [9, 10] considers simply augmenting the target domain data with data from the source domain where an”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “The ability to pass over invariant features of the background is often taken for granted in many supervised tasks with state-of-the-art deep architectures [38].”
The limits and potentials of deep learning for robotics (2018, background): “Some research [84], [110] considers simply augmenting the target domain data with data from the source domain where an alignment exists.”
Moment Matching for Multi-Source Domain Adaptation (2018, neutral): “Dataset Year Images Classes Domains Description Digit-Five - ∼100,000 10 5 digit Office [37] 2010 4,110 31 3 office Office-Caltech [11] 2012 2,533 10 4 office CAD-Pascal [33] 2015 12,000 20 6 animal,vehicle Office-Home [43] 2017 15,500 65 4 office, home PACS [21] 2017 9,991 7 4 animal, stuff Open MI”
Playing for Data: Ground Truth from Computer Games (2016, neutral): “3D object models are also increasingly used for training representations for object detection and object pose estimation [4, 26, 30, 34, 35, 45].”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, methodology): “Rendered views of object CAD models over different (textured) backgrounds have been used as training images for CNN-based object detection [34, 35] and viewpoint estimation [45].”
Deep CORAL: Correlation Alignment for Deep Domain Adaptation (2016, background): “Even though state-of-the-art Deep Convolutional Neural Network features are invariant to low level cues to some degree [2,3,4], Donahue et al. [5] showed that they still are susceptible to domain shift.”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, methodology): “have trouble generalizing to real data because of the change in image statistics [5, 36].”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “mount of annotated data. Synthetic data based adaptation. Early methods use 3D CAD models to improve solutions for pose and viewpoint estimation [161, 162, 163, 164], object and object part detection [165, 166, 167, 168, 169, 170, 171, 172], segmentation and scene understanding [173, 174, 175]. The ”
To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction (2016, neutral): “Learning from synthetic data has a long tradition in computer vision and recently has gained increasing interest [18,25,22,24] due to data hungry deep learning approach.”
VirtualWorlds as Proxy for Multi-object Tracking Analysis (2016, neutral): “Several other works use 3D CAD models for more general object pose estimation [21, 22] and detection [23, 24].”
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints (2015, positive): “is able to learn from generated synthetic examples to perform pose estimation on real-world images. Deep visual representations have been successfully trained from synthetic imagery (Su et al., 2015; Peng et al., 2015) and generally have a greater degree of domain invariance than conventional method”
Modeling Visual Context is Key to Augmenting Object Detection Datasets (2018, background): “If varies from trivial geometrical transformations such as horizontal ﬂipping, cropping with color perturbations, and adding noise to an image [27], to synthesizing new training images [28,29].”