Structured Prediction of Unobserved Voxels from a Single Depth Image

By Michael Firman, Oisin Mac Aodha, Simon Justin Julier, Gabriel J. Brostow. 2016

186 citations recorded by the source corpus.

Building a complete 3D model of a scene, given only a single depth image, is underconstrained. To gain a full volumetric model, one needs either multiple views, or a single view together with a library of unambiguous 3D models that will fit the shape of each individual object in the scene. We hypothesize that objects of dissimilar semantic classes often share similar 3D shape components, enabling a limited dataset to model the shape of a wide range of objects, and hence estimate their hidden geometry. Exploring this hypothesis, we propose an algorithm that can complete the unobserved geometry of tabletop-sized objects, based on a supervised model trained on already available volumetric elements. Our model maps from a local observation in a single depth image to an estimate of the surface shape in the surrounding neighborhood. We validate our approach both qualitatively and quantitatively on a range of indoor object collections and challenging real scenes.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “For scenes, techniques include plane fitting [39], 3D model fitting and retrieval [15, 40] or leverage symmetries [28] and predict 3D semantics [4, 14, 59].”
Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images (2015, background): “Beyond recognition, future works include extending this discriminative learned represention to enable shape completion [29, 6].”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “This idea is also widely used in 3D shape completion from depths or silhouettes [Firman et al., 2016, Rock et al., 2015, Dai et al., 2017].”
OctNetFusion: Learning Depth Fusion from Data (2017, methodology): “We propose OctNetFusion, a deep 3D convolutional neural network that is capable of fusing depth information from (a) Firman [14] (b) Ours (c) Ground Truth”
Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis (2016, methodology): “One of the ﬁrst methods in this space is Voxlets [9].”
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction (2016, methodology): “Inspired by early works that used machine learning to learn a 2D-to-3D mapping for scene understanding [44, 29], data driven approaches have been recently proposed to solve the daunting problem of recovering the shape of an object from just a single image [50, 30, 20, 47] for a given number of objec”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “NYUv2 [31] Real-world 11 RGB-D Mesh/Voxel 1449 795/654 NYUCAD [11] Synthetic 11 RGB-D Mesh/Voxel 1449 795/654 SUNCG [12] Synthetic 11 Depth Mesh/Voxel 45622 139368/470 SemanticKITTI [3] Real-world X 28 Points/RGB Points/Voxel 22 23201/20351 KITTI-360 [19] Real-world X 19 Points Points 11 90960 nuSce”
3D Object Reconstruction from a Single Depth View with Adversarial Learning (2017, background): “[11] trained a random decision forest to predict unknown voxels.”
Pairwise Decomposition of Image Sequences for Active Multi-view Recognition (2016, unknown): “…Rather than modelling an object as a set of views with 2D features, an explicit 3D shape can be learned from reconstruction [37] or provided by CAD models [39], and subsequently matched to from depth images [13], 3D reconstructions [1], or partial reconstructions with shape completion [12, 39].”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, background): “Early approaches addressed the task of scene completion either without predicting semantics [16], thereby not providing a holistic understanding of the scene, or by trying to fit a fixed number of mesh models to the scene geometry [20], which limits the expressiveness of the approach.”
3D Shape Perception from Monocular Vision, Touch, and Shape Priors (2018, unknown): “Recently, there have also been a growing number of papers on completing shapes via their structure and regularities [6] and on learning shape priors in a fully data-driven style [7].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “[63] trained a random decision forest to infer unknown voxels.”
From Point Clouds to Mesh Using Regression (2017, methodology): “To our knowledge, our method is the ﬁrst learning based surface reconstruction, with the exception on the not directly related work of Firman et al. [9], that formulates the prediction of unobserved (occluded) structures as a regression task.”
Anisotropic Convolutional Networks for 3D Semantic Scene Completion (2020, background): “The second dataset is the NYUCAD dataset [5].”
Unsupervised Monocular Depth Estimation with Left-Right Consistency (2016, background): “Finally, while our model estimates per pixel depth, it would be interesting to also predict the full occupancy of the scene [11].”
RGBD Datasets: Past, Present and Future (2016, methodology): “Examples: [21] captures objects, and [30] captures tabletop scenes.”
Pushing the Boundaries of View Extrapolation With Multiplane Images (2019, background): “Finally, a recent line of work [8, 27, 34] focuses on scene shape completion.”