Learning to Reconstruct Shapes from Unseen Classes

By Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Josh Tenenbaum, Bill Freeman, Jiajun Wu. 2018

164 citations recorded by the source corpus.

From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life. Contemporary single-image 3D reconstruction algorithms aim to solve this task in a similar fashion, but often end up with priors that are highly biased by training classes. Here we present an algorithm, Generalizable Reconstruction (GenRe), designed to capture more generic, class-agnostic shape priors. We achieve this with an inference network and training procedure that combine 2.5D representations of visible surfaces (depth and silhouette), spherical shape representations of both visible and non-visible surfaces, and 3D voxel-based representations, in a principled manner that exploits the causal structure of how 3D shapes give rise to 2D images. Experiments demonstrate that GenRe performs well on single-view shape reconstruction, and generalizes to diverse novel objects from categories not seen during training.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology, Generative Models, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

In Generative Models

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “While recent works [74, 76, 79] have applied 3D convolutional neural networks to resolutions up to 128 3 , this is only possible with shallow architectures and small batch sizes, which leads to slow training.”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Another path is the generalization of learning algorithms to novel objects [34, 38, 49].”
Mesh R-CNN (2019, methodology): “To date Pix3D has primarily been used to evalute shape predictions for models trained on ShapeNet, using perfectly cropped, unoccluded image segments [41, 60, 73], or synthetic rendered images of Pix3D models [76].”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Higher resolutions have been used [74, 73, 81] at the cost of limited training batches and slow training or lossy 2D projections [63].”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, methodology): “For shape completion, we use the benchmark provided by GenRe [49], which contains renderings of each shape in ShapeNet from 20 random views.”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, methodology): “Specifically we evaluate 3DR2N2 [7], GenRe [66], Occupancy Networks [41], and Mesh R-CNN [17].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “Wu et al. (2017) and Zhang et al. (2018) use 2.5D supervision during training, i.e., depth maps.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, background): “Higher resolutions (256 3 ) [82, 81, 87] can be achieved at the cost of slow training, or diﬃcult multi-resolution implementations [31, 72, 77].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “Volumetric based approaches partition the space into a regular 3D volume with millions of small voxels [23, 27, 54, 55, 60, 40], and the network predicts if a voxel is on the surface or not.”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “[8, 53] Moreover, our formulation is independent of the choice of world space and the order of input views.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “To achieve high resolution 3D volumetric reconstruction, many recent papers use intermediation, through multiple depth maps, followed by volumetric [38], [46], [51], [92] or point-based [80] fusion.”
Few-Shot Generalization for Single-Image 3D Reconstruction via Priors (2019, background): “Class agnostic models might be more generalizable if they incorporate geometrical constraints [36] or leverage pose information[11].”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “Shape-agnostic priors have been used for 3D reconstruction [2,58,64].”