Unsupervised Learning of 3D Object Categories from Videos in the Wild

By Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, David Novotný. 2021

81 citations recorded by the source corpus.

Our goal is to learn a deep network that, given a small number of images of an object of a given category, reconstructs it in 3D. While several recent works have obtained analogous results using synthetic data or assuming the avail-ability of 2D primitives such as keypoints, we are interested in working with challenging real data and with no manual annotations. We thus focus on learning a model from multiple views of a large collection of object instances. We contribute with a new large dataset of object centric videos suitable for training and benchmarking this class of models. We show that existing techniques leveraging meshes, voxels, or implicit surfaces, which work well for reconstructing isolated objects, fail on this challenging data. Finally, we propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction while obtaining a detailed implicit representation of the object surface and texture, also compensating for the noise in the initial SfM reconstruction that bootstrapped the learning process. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks and on our novel dataset. For additional material please visit: https://henzler.github.io/publication/unsupervised_videos/.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “A recent successful line of work for 3D reconstruction extends NeRF [37] to cross-scene generalization from one or few views by conditioning on image embeddings [20,46, [48].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “Among the methods, some [75, 61, 21, 89, 87, 37, 33, 70, 71] directly aggregate 2D features with MLPs or transformers, while others explicitly construct the 3D feature/cost volume [6, 28, 92, 40], and utilize the voxel feature for decoding density and color.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, methodology): “Warp-conditioned embedding To fix the latter, [26] proposed Warp-Conditioned Embedding (WCE): given a world coordinate point x and a source view Isrc with camera P src, the warp-conditioned embedding zWCE 2 Rz”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Henzler et al. [251] proposed a new neural network called warp-conditioned ray embedding (WCR) that focuses on training a model on multiple views on a large collection of object instances to learn a deep network that reconstructs in 3D given a small number of images.”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, methodology): “Early approaches used NeRF [24] for objects [6,16,18, 22, 27, 40, 49] and scenes [3, 5, 6, 10, 44] 3D reconstrution. pixelNeRF [49] pio-neered the paradigm of predicting pixel-aligned features from images for radiance field reconstruction.”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, background): “Most of these approaches train deep networks specifically for the goal of inferring NeRFs from a small number of views; examples include IBRNet [44], NeRF-WCE [9], Pixel-2”