Aligning 3D models to RGB-D images of cluttered scenes

By Saurabh Gupta, Pablo Arbeláez, Ross B. Girshick, Jitendra Malik. 2015

250 citations recorded by the source corpus.

The goal of this work is to represent objects in an RGB-D scene with corresponding 3D models from a library. We approach this problem by first detecting and segmenting object instances in the scene and then using a convolutional neural network (CNN) to predict the pose of the object. This CNN is trained using pixel surface normals in images containing renderings of synthetic objects. When tested on real data, our method outperforms alternative algorithms trained on real data. We then use this coarse pose estimate along with the inferred pixel support to align a small number of prototypical models to the data, and place into the scene the model that fits best. We observe a 48% relative improvement in performance at the task of 3D detection over the current state-of-the-art [34], while being an order of magnitude faster.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

Cross Modal Distillation for Supervision Transfer (2015, methodology): “We see that our scheme for supervision transfer improves performance from 31.7% achieved when using random initialization to 35.7%, which is more than half way towards what fully supervised pre-training can achieve (38.4%), thereby illustrating the efficacy of our adaptation scheme.”
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (2015, methodology): “Table 3 shows the comparison with the two state-of-the-arts for amodal 3D detection: 3D Sliding Shapes [25] with hand-crafted features, and 2D Depth-RCNN [10] with ConvNets features.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Another related research direction is retrieving similar 3D shapes given a single image, instead of reconstructing the object’s actual geometry [1, 15, 19, 48].”
OctNetFusion: Learning Depth Fusion from Data (2017, background): “This has led to the creation of large databases of real-world 3D content [7, 10, 42, 43], enabling progress in a variety of areas including 3D modeling [28, 54], 3D reconstruction [6, 52], 3D recognition [18, 44] and 3D scene understanding [16, 42].”
Semantic Scene Completion from a Single Depth Image (2016, background): “One possible approach to obtain the complete geometry and semantic labels for a scene is to retrieve and fit instance-level 3D mesh models to the observed depth map [7, 30, 4, 16, 23, 17, 14].”
Marr Revisited: 2D-3D Alignment via Surface Normal Prediction (2016, methodology): “Similar to [15] we consider only those objects which have valid depth pixels for more than 50%.”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, background): “{Farabet, Couprie, Najman, and LeCun} 2013SEDAGHAT et al.: ORIENTATION-BOOSTED VOXEL NETS FOR 3D OBJECT RECOGNITION – BMVC 2017 3 conceptually very similar to feature learning in images. Gupta et al. [12] ﬁts and projects 3D synthetic models into the image plane. 3D convolutional neural networks (CN”
Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics (2017, methodology): “[19] classified coarse object category and pose of objects in a scene by finetuning the AlexNet CNN [28] on depth and color renders from 3D models in ModelNet.”
Instance-Level Segmentation for Autonomous Driving with Deep Densely Connected MRFs (2015, background): “Another line of work matches CAD models to images [2, 20, 10].”
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks (2016, methodology): “For the depth based model we encode the depth using HHA same as [9].”
Learning 3D Object Categories by Looking Around Them (2017, background): “Often the idea is to search for the 3D model in a CAD library that best fits the image [19, 1, 13, 2].”
Domain Randomization and Generative Models for Robotic Grasping (2017, methodology): “Typically, grasps for an object are selected based on sensor data by registering images to a known database of 3D models using a traditional computer vision pipeline [6], [12], [49] or a deep neural network [11], [51].”
Learning 3D Shape Completion from Laser Scan Data with Weak Supervision (2018, unknown): “In [3, 10, 13, 14, 21, 30, 32] shape”
Learning with Side Information through Modality Hallucination (2016, unknown): “Many of these methods do so by introducing new depth representations [19, 29, 32, 38, 36], most recently by adding an additional depth network representation into a convolutional network architecture [15, 14, 35].”
Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications (2017, methodology): “Synthetic data has been successfully used in a range of application domains including prediction of object pose [33, 21, 11], optical flow [6], semantic segmentation for indoor scenes [12, 36], and analysis of image features [2, 15].”
Pattern Recognition (2015, unknown): “ours: wet ours: wet ours: wet ours: dry [9]: wet [22]: wet [15]: wet [9]: snow [22]: dry [15]: dry [9]: dry [22]: wet [15]: dry [9]: snow [22]: dry [15]: dry”
SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth (2016, background): “Recent years have witnessed a huge interest in scene understanding, largely sparked by the seminal work of AlexNet [14] and the increasing popularity of Convolutional Neural Networks (CNNs).”
Policy transfer via modularity and reward guiding (2017, methodology): “An alternative approach would be to train an object pose detector, using one of various image algorithms for such tasks [36], [37], [38], [39].”
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation (2019, background): “Another line of work [34, 19, 9, 28] proposes to first produce 2D object proposals in 2D image and then project the proposal into 3D space to further refine the final 3D bounding box location.”