StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

By Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, J. Park, Ira Kemelmacher-Shlizerman. Computer Vision and Pattern Recognition, 2021

400 citations recorded by the source corpus.

We introduce a high resolution, 3D-consistent image and shape generation technique which we call StyleSDF. Our method is trained on single-view RGB data only, and stands on the shoulders of StyleGAN2 for image generation, while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape. We achieve this by merging a SDF-based 3D representation with a style-based 2D generator. Our 3D implicit network renders low-resolution feature maps, from which the style-based network generates view-consistent, 1024×1024 images. Notably, our SDF-based 3D modeling defines detailed 3D surfaces, leading to consistent volume rendering. Our method shows higher quality results compared to state of the art in terms of visual and geometric quality.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bbfebb10f41254ff88937feafcc653df1cfacb5d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, background): “Generative Adversarial Networks [5] (GANs), are a natural fit, and adversarial training could drop in to replace our diffusion objec-tive with minor changes.”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Here we use the ViT-L/14 CLIP model to compute CLIP R-Precision, but we report results with an alternative CLIP model in Section 5.3.”
CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Current tasks, such as novel view synthesis [32,25,61,12,52] and 3d generation [9,61,8,26,21,11], are trained on in-the-wild image datasets [42,44] which lacks facial dynamics to provide natural geometries.”
Magic3D: High-Resolution Text-to-3D Content Creation (2022, background): “Inspired by the success of neural volume rendering [23], recent works started investing in 3D-aware image synthesis [4, 5, 10, 11, 27, 29, 30, 38], which has the advantage of learning 3D generative models directly from images Ð a more widely accessible resource.”
OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation (2023, background): “Encouraged by NeRF [48], extensive works [6, 7, 25, 27, 52, 55, 66, 67, 85, 93] explore 3Daware image synthesis rather than mesh generation.”
LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “[13] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.”
DreamFusion: Text-to-3D using 2D Diffusion (2022, background): “GANs can learn controllable 3D generators from photographs of a single object category, by placing an adversarial loss on 2D image renderings of the output 3D object or scene (Henzler et al., 2019; Nguyen-Phuoc et al., 2019; Or-El et al., 2022).”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, background): “[51] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher-Shlizerman.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, background): “…Gadelha et al., 2017; Smith & Meger, 2017), triangle meshes (Gao et al., 2022; Pavllo et al., 2020; Chen et al., 2019; Luo et al., 2021) or volumes (Chan et al., 2022; Or-El et al., 2022; Bergman et al., 2022; Skorokhodov et al., 2022) Diffusion models open new possibilities for 3D generation.”
Advances in Neural Rendering (2021, methodology): “Computational cost and quality of geometric reconstructions can be improved by using a surface representation [DYXT21, OELS ∗ 21, XPLD21].”