Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views

By Hao Su, Charles Ruizhongtai Qi, Yangyan Li, Leonidas J. Guibas. 2015

752 citations recorded by the source corpus.

Object viewpoint estimation from 2D images is an essential task in computer vision. However, two issues hinder its progress: scarcity of training data with viewpoint annotations, and a lack of powerful features. Inspired by the growing availability of 3D models, we propose a framework to address both issues by combining render-based image synthesis and CNNs (Convolutional Neural Networks). We believe that 3D models have the potential in generating a large number of images of high variation, which can be well exploited by deep CNN with a high learning capacity. Towards this goal, we propose a scalable and overfit-resistant image synthesis pipeline, together with a novel CNN specifically tailored for the viewpoint estimation task. Experimentally, we show that the viewpoint estimation from our pipeline can significantly outperform state-of-the-art methods on PASCAL 3D+ benchmark.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, methodology): “More recently, 3D models have been used to augment training data to aid transferring deep neural networks between datasets and prevent over-ﬁtting on small datasets for tasks like viewpoint estimation [40] and object detection [42], [29].”
RealPoint3D: Point Cloud Generation from a Single Image with Complex Background (2018, methodology): “For better simulating the real-world scenarios, background textures have been randomly added in the rendered images according to (Su et al. 2015).”
RenderGAN: Generating Realistic Labeled Data (2016, neutral): “The DCNNs are evaluated on the mean Hamming distance (MHD) i.e. the expected value of bits decoded wrong.”
Sim-to-Real Robot Learning from Pixels with Progressive Nets (2016, background): “akes it hard to distinguish between data points drawn from the different domains. These ideas have, however, not yet been tested in the context of control. Demonstrating the difﬁculty of the problem, [10] provides evidence that a simple application of a model trained on synthetic data on the real ro”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “We compare our method with Render for CNN [55].”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “background/lighting variations [30] and initializing the convolution layers with a pretrained ResNet-18 model [18].”
3D Shape Induction from 2D Views of Multiple Objects (2016, methodology): “This may be obtained using a generic view-point estimator such as [30, 28].”
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling (2016, methodology): “…can be used for 3D shape classification [Su et al., 2015a, Sharma et al., 2016, Maturana and Scherer, 2015], 3D shape retrieval [Shi et al., 2015, Su et al., 2015a], and single image 3D reconstruction [Kar et al., 2015, Bansal et al., 2016, Girdhar et al., 2016], mostly with full supervision.”
The limits and potentials of deep learning for robotics (2018, background): “Some research [84], [110] considers simply augmenting the target domain data with data from the source domain where an alignment exists.”
Multi-view 3D Models from Single Images with a Convolutional Network (2015, methodology): “As in Su et al. [26], we randomly sampled the number of light sources from 2 to 4, each with random intensity and at random location.”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, methodology): “We augmented the data similar to [45] by compositing our rendered views over backgrounds sampled from natural images during training, which improved performance.”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, methodology): “Our method is also complementary to existing work [12, 34, 47] that ensures global consistency and can be combined with them.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “itecture [211] for end-to-end action recognition in videos. (Image courtesy C. De Souza). Using synthetic data. Early methods used 3D CAD models to improve solutions for pose and viewpoint estimation [2, 205, 162, 214], object and object part detection [213, 166, 217, 182, 165, 112, 145, 16], segmen”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “[24] used individual objects rendered in front of arbitrary backgrounds with prescribed angles relative to the camera to generate data for learning to predict object pose.”
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (2017, methodology): “by rendering the corresponding 3D model. Recently, 2D object detection methods are used as template matching and augmented for 6D pose estimation, especially with deep learning-based object detectors [28, 23, 16, 29]. Templatebased methods are useful in detecting texture-less objects. However, they ”
Adapting Deep Visuomotor Representations with Weak Pairwise Constraints (2015, neutral): “rmance, however, and state-of-the-art pose estimation methods generally train exclusively on real imagery [12], [13]. Deep visual representations have been successfully trained from synthetic imagery [14], [15] and generally have a greater degree of domain invariance than conventional methods; nonet”
Modeling Visual Context is Key to Augmenting Object Detection Datasets (2018, background): “Regarding image synthesis, recent works such as [31,32,33] build and train their models on purely synthetic rendered 2d and 3d scenes.”
Learning 3D Object Categories by Looking Around Them (2017, methodology): “As viewpoint prediction with deep networks benefits significantly from large training sets [36], we increase the effective size of the training videos by data augmentation.”