GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

By Katja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas Geiger. Neural Information Processing Systems, 2020

989 citations recorded by the source corpus.

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/62d337dbaead376ca042f23d62c0d4b65ec98546

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, background): “Several works aim to train 3D-aware GANs from datasets of 2D images (Chan et al., 2020; Schwarz et al., 2020; Chan et al., 2021; Or-El et al., 2021; Gu et al., 2021; Zhou et al., 2021b).”
Neural RGB-D Surface Reconstruction (2021, background): “NeRF has been extended to handle in-the-wild data with different lighting and occluders [45], dynamic scenes [40, 57], avatars [26], and adapted for generative modeling [5, 66] and image-based rendering [77, 87].”
Unsupervised Learning of Probably Symmetric Deformable 3D Objects From Images in the Wild (2019, methodology): “Some of these methods start from 3D representations [12], [13], [14], [32], [66].”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “Rather than supervising directly on the 3D asset a model generates, these methods supervise the 2D renderings of the generated 3D asset, often using an adversarial loss [3, 4, 38, 48, 64].”
AutoInt: Automatic Integration for Fast Neural Volume Rendering (2020, background): “Coordinate-based networks have also been explored in the context of generative frameworks [7, 9, 20, 39, 40, 52].”
Point-NeRF: Point-based Neural Radiance Fields (2022, background): “They have been extended to achieve dynamic scene capture [26, 38], relighting [3, 5], appearance editing [53], fast rendering [15, 56], and generative models [7, 35, 43].”
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis (2021, background): “Few-shot radiance fields Several works condition NeRF on latent codes describing scene geometry or appearance rather than estimating NeRF per scene [32, 38, 44].”
GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds (2021, background): “Follow-up works have tried to improve the output quality [31, 68], make it faster to train and evaluate [30, 31, 41, 51, 59], extend it to deformable objects [15, 28, 48, 50, 63], account for lighting [9, 6, 37, 57] and compositionality [17, 44, 46, 67], as well as add generative capabilities [11, 5”
Advances in Neural Rendering (2021, background): “Here, some approaches learn the texture in image space [CLX ∗ 21, XPY ∗ 21], while others learn both geometry and texture in 3D [SWZ ∗ 21, SLNG20].”
DreamFusion: Text-to-3D using 2D Diffusion (2022, background): “Many 3D generative approaches have found success in incorporating NeRF-like models in the generative process (Schwarz et al., 2020; Chan et al., 2021b;a; Gu et al., 2021; Liu et al., 2022).”
IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “While NeRF opens up many new research opportunities [30, 36, 43, 45, 55, 58], it must be optimized for each new scene, taking hours or days to converge.”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “However, these models are category-speciﬁc and trained on large datasets of mostly forward-facing scenes [5,20,38,51,66], lacking the ﬂexibility of open-set text-conditional models.”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, methodology): “Several methods modify the architecture of the GAN to enable 3D control [Chan et al. 2022, 2021; Chen et al. 2022; Gu et al. 2022; Pan et al. 2021; Schwarz et al. 2020; Tewari et al. 2022; Xu et al. 2022].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, background): “Conditional NeRFs infer latent codes as pioneered in GRAF [51], piGAN [4], and PixelNeRF [62], as well as the recent CodeNeRF [22], which also optimizes over object poses.”
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields (2021, background): “NeRF has inspired many subsequent works that extend its continuous neural volumetric representation for generative modeling [8, 36], dynamic scenes [23, 33], nonrigidly deforming objects [13, 34], phototourism settings with changing illumination and occluders [26, 43], and reflectance modeling for r”
Editing Conditional Radiance Fields (2021, methodology): “Notice how in the CARLA dataset [17, 63], despite training on only one image per car instance, the model is able to infer the occluded regions of the car.”