Efficient Geometry-aware 3D Generative Adversarial Networks

By Eric Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis and 2 more. Computer Vision and Pattern Recognition, 2021

1,520 citations recorded by the source corpus.

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. We introduce an expressive hybrid explicit implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN generators, such as StyleGAN2, and inherit their efficiency and expressiveness. We demonstrate state-of-the-art 3D-aware synthesis with FFHQ and AFHQ Cats, among other experiments.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7c0a7419114db2209c2f386bc1537e90417cf9d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, background): “In addition, some single-view synthesis methods [59] either allow new generative objects from unseen views or focus on specific objects, e.g., face [4] and human [23].”
3D Neural Field Generation Using Triplane Diffusion (2022, positive): “Similar to Chan et al. [6], we found the sum to be an efficient feature aggregation function, resulting in the following formulation for the triplane architecture: The feature planes and MLP can be jointly optimized to represent the occupancy field of a shape.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “…Gadelha et al., 2017; Smith & Meger, 2017), triangle meshes (Gao et al., 2022; Pavllo et al., 2020; Chen et al., 2019; Luo et al., 2021) or volumes (Chan et al., 2022; Or-El et al., 2022; Bergman et al., 2022; Skorokhodov et al., 2022) Diffusion models open new possibilities for 3D generation.”
RODIN: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion (2022, methodology): “Specifically, we can concatenate the tri-plane features in the channel dimension as in [9], which forms y = (yuv⊕ywu⊕yvw) ∈ RH×W×3C , and employ a well-designed 2D U-Net to model the data distribution through the denoising diffusion process.”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “Encouraged by NeRF [48], extensive works [6, 7, 25, 27, 52, 55, 66, 67, 85, 93] explore 3Daware image synthesis rather than mesh generation.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “Camera simulation: Following recent success in view synthesis and generation [9, 54], we adopt a hybrid volume and neural rendering framework for efficient photorealistic image simulation.”
3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models (2023, background): “The key building block is a neural network which accepts a 3D coordinate as input, and outputs a scalar [Chen and Zhang 2019; Mescheder et al. 2019; Michalkiewicz et al. 2019; Park et al. 2019] or a vector [Chan et al. 2022; Mildenhall et al. 2020; Tang et al. 2022].”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, methodology): “Several authors have proposed to extract 3D models from 2D image generators, originally using GANs [2, 6, 8, 24, 25, 47].”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “Additionally, we are able to generate novel views significantly faster – in practice, stochastic conditioning requires 256 denoising steps to generate each novel view while our method only requires 25, leading to a 10 x improvement in speed.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Although NeRF has become widely used in both 3D reconstruction (Barron et al., 2022; Li et al., 2023d; Chen et al., 2022; Hedman et al., 2021) and generation (Poole et al., 2022; Lin et al., 2023; Chan et al., 2022), optimizing NeRF can be time-consuming.”
Instant neural graphics primitives with a multiresolution hash encoding (2022, methodology): “Parametric input encodings, when used in a generative setting, typically arrange their features in a dense grid which can then be populated by a separate generator network, typically a CNN such as StyleGAN [Chan et al. 2021; DeVries et al. 2021; Peng et al. 2020b].”
Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “We therefore choose to render a feature image in this latent space from a features field [3, 38] represented by a voxel grid in R4×Nx×Ny×Nz .”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, methodology): “Our decomposition method extends the tri-projection in EG3D [6].”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, methodology): “However, these models adopt triplane NeRF [7,44] as the scene representation, which suffers from a limited triplane resolution and expensive volume rendering.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “EG3D [4] partially mitigates this problem by utilizing a dual discriminator.”
Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “Motivated by recent advances in explicit-implicit hybrid scene representations [7, 8], we generalize BEV by accompanying it with two perpendicular planes to construct three cross-planes perpendicular to each other.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Current tasks, such as novel view synthesis [32,25,61,12,52] and 3d generation [9,61,8,26,21,11], are trained on in-the-wild image datasets [42,44] which lacks facial dynamics to provide natural geometries.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, methodology): “Concurrent works [8, 17, 25, 74] adopt two-stage rendering process or smart sampling procedures for highresolution image generation, yet these works still do not model well-defined, view-consistent 3D geometry.”