pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

By Eric Chan, M. Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein. 2020

978 citations recorded by the source corpus.

We have witnessed rapid progress on 3D-aware image synthesis, leveraging recent advances in generative visual models and neural rendering. Existing approaches how-ever fall short in two ways: first, they may lack an under-lying 3D representation or rely on view-inconsistent rendering, hence synthesizing images that are not multi-view consistent; second, they often depend upon representation network architectures that are not expressive enough, and their results thus lack in image quality. We propose a novel generative model, named Periodic Implicit Generative Adversarial Networks (π-GAN or pi-GAN), for high-quality 3D-aware image synthesis. π-GAN leverages neural representations with periodic activation functions and volumetric rendering to represent scenes as view-consistent radiance fields. The proposed approach obtains state-of-the-art results for 3D-aware image synthesis with multiple real and synthetic datasets.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “In contrast to the above-mentioned 2D GANs, the recently developed 3D-aware GANs [74], [75] bridge the gap between 2D images and 3D physical world.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, methodology): “We compare our method with state-of-the-art 3D-aware GANs, e.g ., Pi-GAN [10] and GIRAFFE [42] and EG3D [9], which learn to produce neural radiance ﬁeld from 2D image supervision.”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Coordinate-based networks have also been explored in the context of generative frameworks [7, 9, 20, 38, 39, 50].”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “We base our implementation of this experiment on top of Threestudio [7].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Some works [7, 16, 38, 45, 63] integrate NeRF with GANs to design 3D-aware generators.”
Neural RGB-D Surface Reconstruction (2021, background): “NeRF has been extended to handle in-the-wild data with different lighting and occluders [45], dynamic scenes [40, 57], avatars [26], and adapted for generative modeling [5, 66] and image-based rendering [77, 87].”
StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis (2021, methodology): “, 2020), π-GAN (Chan et al., 2021) and GIRAFFE (Niemeyer & Geiger, 2021b).”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Current tasks, such as novel view synthesis [32,25,61,12,52] and 3d generation [9,61,8,26,21,11], are trained on in-the-wild image datasets [42,44] which lacks facial dynamics to provide natural geometries.”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “Recently, neural implicit representations demonstrated promising results for object geometry representation [8, 22, 24, 32, 34–36, 40, 55, 60, 63, 64], scene completion [6, 17, 37], novel view synthesis [23, 25, 38, 66] and also generative modelling [7, 30, 31, 43].”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, methodology): “In contrast, globally conditioned methods [3, 11, 1, 2] first infer a global representation that is invariant to the number of context views and subsequently discard the observations.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, positive): “The radiance field function F Θ is often implemented with multi-layer perceptrons (MLPs) with positional encodings [43,27] and periodic activations [2].”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, methodology): “Neural rendering with implicit representations has become a widely-used technique for solving many vision and graphics tasks ranging from view synthesis [5, 15, 25], to re-lighting [12,13], to pose and shape estimation [17,21,31], to 3D-aware image synthesis and editing [3, 11, 23], to modeling dyna”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “NeRF [10], along with existing 3D-aware GANs like PiGAN [2], rely on hierarchical sampling strategy for obtaining more samples near the surface.”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, background): “Its simplicity and impressive results inspires a lot of following works, including in-the-wild reconstruction [30, 51], lighting and material estimation [3,47,67], generation [5,17,21, 42, 52, 53], human rendering [44, 50, 57, 68, 69] and so on.”