Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild

By Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi. 2019

338 citations recorded by the source corpus.

We propose a method to learn 3D deformable object categories from raw single-view images, without external supervision. The method is based on an autoencoder that factors each input image into depth, albedo, viewpoint and illumination. In order to disentangle these components without supervision, we use the fact that many object categories have, at least in principle, a symmetric structure. We show that reasoning about illumination allows us to exploit the underlying object symmetry even if the appearance is not symmetric due to shading. Furthermore, we model objects that are probably, but not certainly, symmetric by predicting a symmetry probability map, learned end-to-end with the other components of the model. Our experiments show that this method can recover very accurately the 3D shape of human faces, cat faces and cars from single-view images, without any supervision or a prior shape model. On benchmarks, we demonstrate superior accuracy compared to another method that uses supervision at the level of 2D image correspondences.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In 3D Vision & Pose

Citations in the corpus

Multiview Compressive Coding for 3D Reconstruction (2023, background): “Reminiscent of generalized cylinders [40], some introduce object-specific priors via category-specific 3D templates [25,29,31], pose [42] or symmetries [73].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, methodology): “Other self-supervised methods do not leverage face-domain-specific knowledge, which makes them applicable to general objects, but also limits the reconstruction quality [79, 95].”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, neutral): “Exploiting symmetries and reasoning about lighting [71], compose appearance, shape and lighting.”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, background): “Various 3D representations have been considered for the task, including voxels [18, 62], meshes [23, 6, 14, 17, 40], point clouds [59], a depth map [65], and implicit functions [50, 61].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “These methods are typically structured as auto-encoders [92, 36, 44] or generators [7, 81] with explicit 3D decomposition under the hood.”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, methodology): “…using multiple images, there are also learning-based approaches [Niklaus et al. 2019; Shih et al. 2020; Tucker and Snavely 2020; Wiles et al. 2020; Wu et al. 2020; Zhou et al. 2016], which allow for single-image novel view synthesis at inference time by learning a prior over a collection of…”
Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs (2020, methodology): “Finally, Π applies this warp to the illuminated texture J to obtain the repro-jected is the backward warping of η d,v , which is achieved via a differentiable renderer Kato et al. (2018) in the same way as Wu et al. (2020).”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “Similarly, [62] does not require pose supervision, but it has been demonstrated only for limited viewpoint variations.”