Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views

By Francisco Massa, Bryan C. Russell, Mathieu Aubry. 2015

100 citations recorded by the source corpus.

This paper presents an end-to-end convolutional neural network (CNN) for 2D-3D exemplar detection. We demonstrate that the ability to adapt the features of natural images to better align with those of CAD rendered views is critical to the success of our technique. We show that the adaptation can be learned by compositing rendered views of textured object models on natural images. Our approach can be naturally incorporated into a CNN detection pipeline and extends the accuracy and speed benefits from recent advances in deep learning to 2D-3D exemplar detection. We applied our method to two tasks: instance detection, where we evaluated on the IKEA dataset [36], and object category detection, where we out-perform Aubry et al. [3] for "chair" detection on a subset of the Pascal VOC dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Papier-Mache Approach to Learning 3D Surface Generation (2018, neutral): “Non-parametric approaches retrieve shapes from a large corpus [2, 20, 23], but require having an exact instance in the corpus.”
Playing for Data: Ground Truth from Computer Games (2016, neutral): “3D object models are also increasingly used for training representations for object detection and object pose estimation [4, 26, 30, 34, 35, 45].”
Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization (2016, background): “, and Girshick} 2016{Zagoruyko, Lerer, Lin, Pinheiro, Gross, Chintala, and Doll{á}r} 2016{Cinbis, Verbeek, and Schmid} 20152 S. GIDARIS AND N. KOMODAKIS: ATTEND REFINE REPEAT exemplar 2D-3D detection [30], visual semantic role labelling [17], caption generation [24] or visual question answering [37]”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, methodology): “[31] use real and synthetic images for 2D-3D alignment.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, methodology): “mount of annotated data. Synthetic data based adaptation. Early methods use 3D CAD models to improve solutions for pose and viewpoint estimation [161, 162, 163, 164], object and object part detection [165, 166, 167, 168, 169, 170, 171, 172], segmentation and scene understanding [173, 174, 175]. The ”
Learning 3D Object Categories by Looking Around Them (2017, background): “Note that apart from [1], there exist newer works that tackle the problem of single-view 3D reconstruction [4, 6], However these were not considered due to their requirement of renderable mesh models which are not available in our supervision setting.”
3D-FUTURE: 3D Furniture Shape with TextURE (2020, methodology): “unified constrained embedding space via adaptation techniques such as weight-sharing constraints, metric learning, and distance matching (Li et al., 2015; Aubry et al., 2014; Lee et al., 2018; Massa et al., 2016; Tasse and Dodgson, 2016; Girdhar et al., 2016).”
End-to-End 3D Face Reconstruction with Deep Neural Networks (2017, background): “Three-dimensional information, being a strong prior invariant to view perspectives, has been demonstrated beneficial in different computer vision applications [29, 9, 11, 17, 18, 15].”
Transfer of View-manifold Learning to Similarity Perception of Novel Objects (2017, result): “As shown in many literatures (Massa et al., 2015; Aubry & Russell, 2015), features from different layers sometimes perform differently for a given task.”
Rethinking Reprojection: Closing the Loop for Pose-Aware Shape Reconstruction from a Single Image (2017, methodology): “Notable efforts that use synthetic imagery for training deep models to estimate pose or reconstruct 3D shape include [19, 6, 1, 4, 11, 17, 20, 18].”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, methodology): “Deep-learning methods for viewpoint estimation follow roughly three different paths: direct estimation of Euler angles [55, 50, 33, 24, 62, 63], template-based matching [20, 32, 51], and keypoint detection relying on 3D bounding box corners [39, 52, 13, 34, 36] or semantic keypoints [35, 65].”
Deep representation of industrial components using simulated images (2017, methodology): “As a workaround solution, domain adaptation layers in CNNs are often trained using both real and synthetic data [19], [36].”
Semantic Multi-view Stereo: Jointly Estimating Objects and Voxels (2017, methodology): “For indoor scenes, recent convolutional neural networks (CNN) together with large annotated 3D databases such as ModelNet [37] and ShapeNet [7] produce compelling results for object class detection and rough pose estimation [8, 14, 23, 32].”
Synthesizing Training Images for Boosting Human 3D Pose Estimation (2016, background): “We systematically explore these issues in this paper, with an emphasis on driving the synthesis procedure by real data and bridging the domain gap between synthetic and real data.”