Pixels, Voxels, and Views: A Study of Shape Representations for Single View 3D Object Shape Prediction

By Daeyun Shin, Charless C. Fowlkes, Derek Hoiem. 2018

128 citations recorded by the source corpus.

The goal of this paper is to compare surface-based and volumetric 3D object shape representations, as well as viewer-centered and object-centered reference frames for single-view 3D shape prediction. We propose a new algorithm for predicting depth maps from multiple viewpoints, with a single depth or RGB image as input. By modifying the network and the way models are evaluated, we can directly compare the merits of voxels vs. surfaces and viewer-centered vs. object-centered for familiar vs. unfamiliar objects, as predicted from RGB or depth images. Among our findings, we show that surface-based methods outperform voxel representations for objects from novel classes and produce higher resolution outputs. We also find that using viewer-centered coordinates is advantageous for novel objects, while object-centered representations are better for more familiar objects. Interestingly, the coordinate frame significantly affects the shape representation learned, with object-centered placing more importance on implicitly recognizing the object category and viewer-centered producing shape representations with less dependence on category recognition.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “Another way of representing 3D shapes is to use a set of multi-view depth images [Soltani et al., 2017, Shin et al., 2018, Jayaraman et al., 2018].”
Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation (2018, neutral): “Multi-View Representation Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D [17, 26, 42, 32, 12, 39, 34], such as decomposing image silhouettes [23? ], Light Field Descriptors [2], and 2D panoramic mapping [38].”
What Do Single-View 3D Reconstruction Networks Learn? (2019, methodology): “We follow standard procedure and measure shape similarity with the mean Intersection over Union (mIoU) metric, aggregating predictions within semantic classes [4, 8, 13, 37, 42, 46, 55].”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “Similarly, view-wise modeling is a common theme in monocular 3D reconstruction works [57, 113, 127, 141] and in view synthesis works [151].”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Another path is the generalization of learning algorithms to novel objects [34, 38, 49].”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “PixelNeRF operates in view-space, which has been shown to allow better reconstruction of unseen object categories in [37, 2], and discourages the memorization of the training set [41].”
Image2Mesh: A Learning Framework for Single Image 3D Reconstruction (2017, background): “3D shapes can be efficiently represented by polygon meshes to encode both geometrical and topological information [41].”