StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

By Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt. 2021

633 citations recorded by the source corpus.

We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

GAN Inversion: A Survey (2021, background): “In contrast to the above-mentioned 2D GANs, the recently developed 3D-aware GANs [75], [76] bridge the gap between 2D images and 3D physical world.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “Method Application Representation Supervision Textured mesh Arbitrary topology OccNet [43] 3D generation Implicit 3D ✗ ✓ PointFlow [68] 3D generation Point cloud 3D ✗ ✓ Texture3D [53] 3D generation Mesh 2D ✓ ✗ StyleNerf [25] 3D-aware NV Neural field 2D ✗ ✓ EG3D [8] 3D-aware NV Neural field 2D ✗ ✓ Pi”
Efficient Geometry-aware 3D Generative Adversarial Networks (2021, background): “Although prior work has proposed learning the pose distribution on the ﬂy [49], others have noticed such methods can diverge [18], so it would be fruitful to explore this direction further.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[14] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt.”
Zero-Shot Text-Guided Object Generation with Dream Fields (2021, methodology): “However, these models are category-speciﬁc and trained on large datasets of mostly forward-facing scenes [5,20,38,51,66], lacking the ﬂexibility of open-set text-conditional models.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “3.3)”
3D Neural Field Generation Using Triplane Diffusion (2022, neutral): “For example, concurrently developed single-latent approaches [2,14] generate a global latent for conditioning the neural field, relying on a 3D decoder to transform the scene representation from 1D to 3D without directly performing 3D diffusion.”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, background): “Several works aim to train 3D-aware GANs from datasets of 2D images (Chan et al., 2020; Schwarz et al., 2020; Chan et al., 2021; Or-El et al., 2021; Gu et al., 2021; Zhou et al., 2021b).”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, positive): “Leveraging various successful 2D GAN architectures [4, 23, 37, 39–42, 112], 3D GANs [5, 6, 22 , 24 , 65 , 67 , 77 , 82 , 87 , 88 , Some concurrent works, such as LGM [92], AGG [103], and Splatter Image [90], also use 3D Gaussians in a feed-forward model.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “2.1 3D G ENERATION Following the success of generative models on 2D images using GANs (Goodfellow et al., 2014; Karras et al., 2019; Gu et al., 2022; Kang et al., 2023) and diffusion models (Rombach et al., 2021; Podell et al., 2023; Ho et al., 2020; Song et al., 2021; Saharia et al., 2022), people…”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Recent NeRF-GAN methods [4, 15, 22, 24, 47, 58] learn to generate NeRFs for specific categories like human or cat faces.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, background): “We compare our method with state-of-the-art 3D-aware GANs, e.g ., Pi-GAN [10] and GIRAFFE [41] and EG3D [9], which learn to produce neural radiance field from 2D image supervision.”