Exploring Invariances in Deep Convolutional Neural Networks Using Synthetic Images

By Xingchao Peng, Baochen Sun, Karim Ali, Kate Saenko. 2014

59 citations recorded by the source corpus.

Crowdsourced 3D CAD models are becoming easily accessible online, and can potentially generate an infinite number of training images for almost any object category. We show that adapting contemporary Deep Convolutional Neural Net (DCNN) models to such data can be effective, especially in the few-shot regime where none or only a few annotated real images are available, or where the images are not well matched to the target domain. Little is known about the degree of realism necessary to train models with deep features on CAD data. In a detailed analysis, we use synthetic images to probe DCNN invariance to object-class variations caused by 3D shape, pose, and photorealism, with surprising findings. In particular, we show that DCNNs used as a fixed representation exhibit a large amount of invariance to these factors, but, if allowed to adapt, can still learn effectively from synthetic data. These findings guide us in designing a method for adaptive training of DCNNs using real and synthetic data. We show that our approach significantly outperforms previous methods on the benchmark PASCAL VOC2007 dataset when learning in the fewshot scenario, and outperform training with real data in a domain shift scenario on the Office benchmark.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning a Predictable and Generative Vector Representation for Objects (2016, methodology): “Among a large body of works trying to infer 3D representations from images, our approach is most related to a group of works using renderings of 3D CAD models to predict properties such as object viewpoint [33] or class [32], among others [10, 25, 31].”
Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views (2015, methodology): “While both [27] and our work connect synthetic images with CNN, they are fundamentally different in task, approach and result.”
Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories (2016, background): “While benchmark scores for many CNN intrinsics models [21, 34, 35, 20, 22] are improving, the visual quality of results remains poor, compared with those from traditional approaches based on hand-crafted features and multitudes of priors [6].”
What Is Holding Back Convnets for Detection? (2015, background): “Some of the strategies considered include video game renderings [41] (aiming at photo-realism), CAD model wire-frame renderings [34,25] (focusing on object boundaries), texture-mapped CAD models [29,23], or augmenting the training set by subtle deformations of the positive samples [7,26].”
Rendering of Eyes for Eye-Shape Registration and Gaze Estimation (2015, background): “[16] demonstrated the importance of fine-tuning models initially trained on synthetic data on real data to increase performance.”
Towards Adapting Deep Visuomotor Representations from Simulated to Real Environments (2015, methodology): “In those methods the learned adaptation was a kernelized transformation over a fixed representation.”
Context Augmentation for Convolutional Neural Networks (2017, methodology): “Recently, Peng et al. [19] use 3D CAD models to create a synthetic dataset for object detection.”
iLab-20M: A Large-Scale Controlled Object Dataset to Investigate Deep Learning (2016, background): “We also discuss how our analyses can lead the field to develop more efficient deep learning methods.”
Synthetic Data for Deep Learning (2019, background): “ing visual feature sets to variation in position, scale, pose, and illumination, while Kaneva et al. [306] used a photorealistic virtual environment to evaluate image feature descriptors. Peng et al. [444], Pepik et al. [447], and Aubry and Russell [30] used synthetic data to study the properties of”
RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints (2016, background): “ signiﬁcance of object pose estimation from a single-view image is also discussed in recent publications that synthesize 3D CAD models of objects into real images for learning real object recognition [16,17,13,18,19,20]. Papon and Schoeler [16] learned a CNN that takes the normal, intensity, and dep”