What Do Single-View 3D Reconstruction Networks Learn?

By Maxim Tatarchenko, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, Thomas Brox. Computer Vision and Pattern Recognition, 2019

502 citations recorded by the source corpus.

Convolutional networks for single-view object reconstruction have shown impressive performance and have become a popular subject of research. All existing techniques are united by the idea of having an encoder-decoder network that performs non-trivial reasoning about the 3D structure of the output space. In this work, we set up two alternative approaches that perform image classification and retrieval respectively. These simple baselines yield better results than state-of-the-art methods, both qualitatively and quantitatively. We show that encoder-decoder methods are statistically indistinguishable from these baselines, thus indicating that the current state of the art in single-view object reconstruction does not actually perform reconstruction but image classification. We identify aspects of popular experimental procedures that elicit this behavior and discuss ways to improve the current state of research.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/b05fb9a7d5e3b596325dec6795ce49ec3ac14907

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “As outlined in the recent paper of Tatarchenko et al. [44], deep learning-based reconstruction methods are biased towards recognition and retrieval.”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “While these approaches arguably perform classification and not actual 3D reconstruction [129], all-in-all, they work only in very constrained setups, usually on ShapeNet [14] and have trouble generalizing to natural scenes with non object-centric views [187].”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Before the prosperity of generative models used in 3D reconstruction, there are many works [19, 20, 33, 37, 68] that reconstruct 3D shapes from single-view images by regression [37] or retrieval [68], which have difficulty in generalizing to real data or new categories.”
Multi-view 3D Reconstruction with Transformers (2021, background): “Compared with IoU, F-score [19, 27] explicitly evaluates the distance between object surfaces, which is more interpretable.”
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation (2022, unknown): “We use 3D IoU, Chamfer Distance (CD), and F-score@1% [28] as the metrics to measure the performance across different methods.”
Local Deep Implicit Functions for 3D Shape (2019, methodology): “We evaluate shape reconstruction results with mean intersection-over-union (IoU) [24], mean Chamfer distance [24], and mean F-Score [38] at τ = 0 .”
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation (2022, methodology): “We use Chamfer Distance (CD), and F-score@1% [44] as evaluation metrics.”
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction (2019, methodology): “Table 2: F-Score for varying thresholds (% of reconstruction volume side length, same as [34]) on all categories.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “• On 3D reconstruction tasks, simple clustering and retrieval in the embedding space outperforms state of the art reconstruction networks [149].”
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation (2019, background): “Even though with impressive success, most deep models adopt an encoder-decoder framework, and it is arguable if they perform shape generation or shape retrieval [42].”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “We hypothesize this occurs because 1) networks learn an overly strong prior on x-y-z point coordinates damaging the invariance to articulation, and 2) the shape encoding vector lacks 3D structure, resulting in decodings that look more like classification into shape prototypes [66] rather than contin”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, methodology): “Finally, we compute the F-score [40] with a threshold of 0 .”
ABO: Dataset and Benchmarks for Real-World 3D Object Understanding (2021, background): “This has led to a variety of 3D reconstruction methods that work well on clear-background renderings of synthetic objects [13, 24, 48, 67] but do not necessarily generalize to real images, new categories, or more complex object geometries [5, 6, 60].”
ShapeFormer: Transformer-based Shape Completion via Sparse Representation (2022, methodology): “For the low ambiguity setting, we use Chamfer L2 Distance (CD) and F-score%1 (F1) [59] to measure how accurate the completion is; this is similar to the previous setup [51].”
Self-supervised Learning of 3D Objects from Natural Images (2019, background): “Another path is the generalization of learning algorithms to novel objects [34, 38, 49].”
3D-FUTURE: 3D Furniture Shape with TextURE (2020, background): “Metrics are IoU (%), CD (×10−3, computed on 2,048 points) and F-score (thresholds is 1%, the reconstruction volume side length defined in (Tatarchenko et al., 2019)).”
Few-Shot Generalization for Single-Image 3D Reconstruction via Priors (2019, background): “demonstrated that single view 3D reconstruction models tend to memorize and retrieve similar shapes from the training set; an indication of overfitting [18].”
PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers (2021, methodology): “We also follow [34] to adopt F-Score as another evaluation metric.”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “PixelNeRF operates in view-space, which has been shown to allow better reconstruction of unseen object categories in [37, 2], and discourages the memorization of the training set [41].”