Learning 3D Object Categories by Looking Around Them

By David Novotný, Diane Larlus, Andrea Vedaldi. 2017

88 citations recorded by the source corpus.

Traditional approaches for learning 3D object categories use either synthetic data or manual supervision. In this paper, we propose a method which does not require manual annotations and is instead cued by observing objects from a moving vantage point. Our system builds on two innovations: a Siamese viewpoint factorization network that robustly aligns different videos together without explicitly comparing 3D shapes; and a 3D shape completion network that can extract the full shape of an object from partial observations. We also demonstrate the benefits of configuring networks to perform probabilistic predictions as well as of geometry-aware data augmentation schemes. We obtain state-of-the-art results on publicly-available benchmarks.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Learning to Reconstruct Shapes from Unseen Classes (2018, background): “…progress in recent years, mostly with voxel representations [Choy et al., 2016, Girdhar et al., 2016, Häne et al., 2017, Kar et al., 2015, Novotny et al., 2017, Rezende et al., 2016, Tatarchenko et al., 2016, Tulsiani et al., 2017, Wu et al., 2016, 2017, 2018, Zhu et al., 2018, Yan et…”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “With the development of large-scale shape repositories like ShapeNet [5] and methods like deep convolutional networks, researchers have built more scalable and eﬃcient models in recent years [7,13,18,27,36,38,48,52,56,57,62].”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Using deep convolutional networks, researchers have made significant progress in recent years [9, 17, 21, 29, 41, 43, 56, 59, 60, 62, 66, 52, 61].”
Lightweight Probabilistic Deep Networks (2018, background): “(18) are not independent, unlike in other recent work employing probabilistic outputs [42].”
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth (2019, background): “Our loss relaxation by learning an optimal clustering margin shows some similarites with [20, 9], where they integrate the aleatoric uncertainty into the loss function.”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “[42, 43] canonically align point clouds by only supervising with relative pose, but only learn a shape model.”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, background): “[47, 48] canonically align point clouds of object-categories in an unsupervised fashion, but do not reason about colour.”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, background): “Rezende et al. (2016), Yan et al. (2016), Gadelha et al. (2016) and Novotný et al. (2017) learn 3D shapes by using multiple views and approximately differentiable rendering mechanisms.”
RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints (2016, background): “[22] proposed a viewpoint factorization network that utilizes relative pose changes within each sequence to align objects in videos in an unsupervised manner.”
C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion (2019, methodology): “[27] proposed the first purely image-driven method for single-view reconstruction of rigid object categories.”
MegaDepth: Learning Single-View Depth Prediction from Internet Photos (2018, methodology): “[26] trained a network on 3D models derived from SfM+MVS on videos to learn 3D shapes of single objects.”
Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model Using Deep Non-Rigid Structure From Motion (2019, background): “multiple views of the same object [45], videos [29] or some pre-existing 3D mesh representation that is the starting point for further disentanglement [13, 35, 53, 40] or self-supervision [56].”
VGGT: Visual Geometry Grounded Transformer (2025, unknown): “AUC is the area under the accuracy-threshold curve of the minimum values between RRA and RTA across varying thresholds.”
Relaxed Softmax: Efficient Confidence Auto-Calibration for Safe Pedestrian Detection (2018, background): “Novotny et al. [20] use network uncertainty as a mean to learn 3D geometry predictions from videos in an unsupervised manner.”
Self-Supervised Learning of Geometrically Stable Features Through Probabilistic Introspection (2018, background): “In order to do so, inspired by [26, 16], the network is modified to compute an additional scalar value σx u ∈ R + for each pixel expressing uncertainty about the quality of the descriptor extracted at u and its consequent ability to establish a reliable match.”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Besides using 3D ground truth directly, authors have considered using videos [2, 68, 40, 59] and stereo pairs [19, 36].”