Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

By Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, Ping Luo. 2020

121 citations recorded by the source corpus.

Natural images are projections of 3D objects on a 2D image plane. While state-of-the-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures. And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images? To answer these questions, in this work, we present the first attempt to directly mine 3D geometric clues from an off-the-shelf 2D GAN that is trained on RGB images only. Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner. The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold. The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes (e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cats, cars, and buildings. The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation. We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation. Our code and models will be released at this https URL.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “common image editing applications, in the last few months, GAN inversion techniques have been widely introduced to many other tasks, such as 3D reconstruction [132], [133],”
Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation (2020, background): “manipulation [15], [39], [40], [41], [42], [43], but is restricted to synthetic images of the GAN itself or real images of limited complexity, e.”
GAN-Supervised Dense Visual Alignment (2021, methodology): “[65] Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo.”
Towards Open-World Text-Guided Face Image Generation and Manipulation (2021, methodology): “To generate 3D shapes directly from text or sound, we can easily integrate our method with a concurrent shape reconstruction method [74] for the reason that we share the same latent space of a pretrained GAN model.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “[48] Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo.”
Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, background): “[30] attempts to reconstruct the 3D shape using pretrained 2D GANs.”
Ensembling with Deep Generative Views (2021, background): “Apart from linear edit directions, recent works [67, 36] learn latent editing operations that control the 3D appearance of generated images.”
Labels4Free: Unsupervised Segmentation using StyleGAN (2021, background): “Examples include various image and video editing tasks, image inpainting [13, 47, 46, 41], local image editing [42, 2], low bit-rate video conferencing [40], image super resolution [28, 12], image colorization [4, 27], and extracting 3D models [30].”