HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

By Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-Liang Yang. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2019

554 citations recorded by the source corpus.

We propose a novel generative adversarial network (GAN) for the task of unsupervised learning of 3D representations from natural images. Most generative models rely on 2D kernels to generate images and make few assumptions about the 3D world. These models therefore tend to create blurry images or artefacts in tasks that require a strong 3D understanding, such as novel-view synthesis. HoloGAN instead learns a 3D representation of the world, and to render this representation in a realistic manner. Unlike other GANs, HoloGAN provides explicit control over the pose of generated objects through rigid-body transformations of the learnt 3D features. Our experiments show that using explicit 3D features enables HoloGAN to disentangle 3D pose and identity, which is further decomposed into shape and appearance, while still being able to generate images with similar or higher visual quality than other generative models. HoloGAN can be trained end-to-end from unlabelled 2D images only. Particularly, we do not require pose labels, 3D shapes, or multiple views of the same objects. This shows that HoloGAN is the first generative model that learns 3D representations from natural images in an entirely unsupervised manner.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8ccb88958358ea59bdc9b76f660b01c8f631b2c0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, methodology): “These methods [9,18,27,32,44,49–52 Most inline with our work are methods that use implicit neural radiance ﬁelds for 3D-aware image and geometry generation (GRAF [61] and Pi-GAN [9]).”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, methodology): “Related work has been successful at being view consistent [4, 58, 59] or modeling pose-appearance correlations [47,49], but cannot achieve both simultaneously.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, methodology): “By introducing differentiable 3D projection and using trainable layers that operate in the 3D and 2D feature space, several recent methods [4, 18, 42, 43, 56, 62] are able to model the geometry and appearance of 3D scenes from 2D images.”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, unknown): “Earlier works addressing this problem primarily leverage Generative Adversarial Networks (GANs) [3, 4, 12–14, 18, 33, 34, 46, 73].”
Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation (2021, methodology): “Instead of post-processing, studies have also been given to achieve disentanglement by incorporating extra regularization [7, 10, 30, 32, 34, 40, 48] in GAN training.”
Advances in Neural Rendering (2021, methodology): “Note that some of the methods [NG21b, NPLT ∗ 19] discussed in Section 4.2 can be used for scene editing as well.”
BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images (2020, methodology): “Recent work on unsupervised disentangled representations using GANs [9, 24, 38] relaxes this need for labels.”
pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis (2020, methodology): “The learned representations can be used to render view-consistent images from new camera poses [41, 54, 18].”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, background): “2D-supervised 3D GANs pioneered [35, 43, 64] the approach of training 3D generative models using only unstructured 2D images, and promise greater scalability in terms of data.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “Inspired by the success of neural volume rendering [25], recent works started investing in 3D-aware image synthesis [4, 5, 10, 11, 29, 31, 32, 40], which has the advantage of learning 3D generative models directly from images — a more widely accessible resource.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “The evolution of these techniques, from Generative Adversarial Networks (GANs)[3, 9, 10, 24, 25, 27] to diffusion-based frameworks[33], marks a notable progression.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Similarly, several GAN works [3, 11, 25, 36, 38, 103, 127, 127, 142, 149] proposed complex GAN architectures for facial”
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (2019, background): “Some prior work infers voxel grid representations of 3D scenes from images [6, 8, 9] or uses them for 3D-structure-aware generative models [10, 36].”
Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction (2021, background): “neural networks are trained to approximate the rendering function [44, 56].”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “Neural Representations: Besides coordinate-based models, neural representations are commonly used for 3D modeling and rendering [54], ranging from voxel grids [24, 29, 38, 41, 49], neural textures [55], and point clouds [1].”
Pix2Shape: Towards Unsupervised Learning of 3D Scenes from Images Using a View-Based Representation (2020, methodology): “Compared to Rezende et al. (2016), our model can learn to represent more complex synthetic indoor scenes composed of multiple ShapeNet (Chang et al. 2015) objects and, while we do not address real image inputs (i.e., as HoloGAN), we can infer explicit geometry for visible surfaces from each given vi”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, unknown): “Leveraging various successful 2D GAN architectures [4, 23, 37, 39–42, 112], 3D GANs [5, 6, 22 , 24 , 65 , 67 , 77 , 82 , 87 , 88 , Some concurrent works, such as LGM [92], AGG [103], and Splatter Image [90], also use 3D Gaussians in a feed-forward model.”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, methodology): “Other controllable neural modeling methods [Chen and Zhang 2019; Mescheder et al. 2019; Nguyen-Phuoc et al. 2019, 2020; Park et al. 2019a] employ traditional implicit geometry representations such as the signed distance field (SDF) and occupancy field (OF) but under neural network approximations.”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, methodology): “Voxel-based GANs (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; 2020) lack fine details in the output due to resolution restriction.”