3D ShapeNets for 2.5D Object Recognition and Next-Best-View Prediction

By Zhirong Wu, Shuran Song, Aditya Khosla, Xiaoou Tang, Jianxiong Xiao. 2014

127 citations recorded by the source corpus.

3D shape is a crucial but heavily underutilized cue in today's computer vision systems, mostly due to the lack of a good generic shape representation. With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft Kinect), it is becoming increasingly important to have a powerful 3D shape representation in the loop. Apart from category recognition, recovering full 3D shapes from view-based 2.5D depth maps is also a critical part of visual understanding. To this end, we propose to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically. It naturally supports joint object recognition and shape completion from 2.5D depth maps, and it enables active object recognition through view planning. To train our 3D deep learning model, we construct ModelNet -- a large-scale 3D CAD model dataset. Extensive experiments show that our 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, neutral): “n Table3, indicating the variability in dataset sizes, in graph sizes, and in the availability of labels. Following [35], we perform 10-fold cross-validation 6 Model ModelNet10 ModelNet40 3DShapeNets [40] 83.5 77.3 MVCNN [38] — 90.1 VoxNet [26] 92 83 ORION [1] 93.8 — SubvolumeSup [30] — 86.0 (89.2) ”
Lifting Object Detection Datasets into 3D (2015, background): “An additional potentially powerful direction for future work is feature learning, in particular to improve the ranking of reconstructions, perhaps using one of the large collections of CADmodels available online [65].”
Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views (2015, methodology): “Using data from these repositories, [38] built a dataset of ∼130K models from over 600 categories.”
Semantically-enriched 3D models for common-sense knowledge (2015, methodology): “We use a 3D model size estimation method designed to propagate physical size priors between models in 3D scenes [20] to obtain physical sizes for our corpus.”
Learning Image Representations Tied to Egomotion from Unlabeled Video (2017, methodology): “We now describe our method for this task, similar in spirit to [51].”
Synthetic Data for Deep Learning (2019, methodology): “They have developed a synthetic data generator and trained on 7000 randomly generated scenes with ≈60K instances of 2842 pose-aligned models from the ModelNet10 dataset [649], showing excellent results in transfer to real test data.”
Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds (2018, methodology): “Translation of Point-Clouds into voxels or image stacks by using Convolutional Neural Networks (CNN) [2] [3] [4] [6] [8] [9] [25] [26] 3.”
SUN RGB-D: A RGB-D scene understanding benchmark suite (2015, background): “There are many interesting works on RGB-D scene understanding, including semantic segmentation [52, 48, 18] object classification [68], object detection [58, 19, 61], context reasoning [37], mid-level recognition [31, 30], and surface orientation and room layout estimation [13, 14, 73].”
Complex-YOLO: Real-time 3D Object Detection on Point Clouds (2018, methodology): “oud processing using Multi-Layer-Perceptrons [5] [10] [11] [23] [24] 2.Conversion of Point-Clouds into voxels or image stacks by using Convolutional Neural Networks (CNN) [2] [3] [4] [6] [8] [9] [25] [26] 3.Combined fusion approaches [2] [7] Complex-YOLO: Real-time 3D Object Detection on Point Cloud”
Aligning 3D models to RGB-D images of cluttered scenes (2015, methodology): “For synthetic data we use the collection of aligned models made available by Wu et al .”
Self-Supervised Deep Learning on Point Clouds by Reconstructing Space (2019, methodology): “In line with previous approaches, we evaluate our performance on the object classification problem using the ModelNet dataset [35], which contains CAD models from different categories of man-made objects.”
Inferring 3D Object Pose in RGB-D Images (2015, methodology): “We use 3D models from ModelNet [38] to train the network.”
SceneNet: An annotated model generator for indoor scene understanding (2016, methodology): “Existing annotated datasets limit their focus to 3D objects rather than scenes [5], [6], or images rather than videos [7], [8].”
Sliding Shapes for 3D Object Detection in Depth Images (2014, methodology): “As future work, we plan to investigate how to combine with RGB-based detection, and learn the 3D features automatically from data [85], as well as exploring context information in 3D [86].”
Shape completion enabled robotic grasping (2016, methodology): “Prior work for shape completion includes [6][3]’s use of a deep belief network and Gibbs sampling for 3D shape reconstruction.”