Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

By Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, Daniel Cohen-Or. Computer Vision and Pattern Recognition, 2022

615 citations recorded by the source corpus.

Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naive solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/793939b83e10903f58d8edbb7534963df627a1fe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “A typical strategy is to use image-based 2D generative models ( e.g ., Imagen [41], Stable Diffusion (SD) [39]) to provide a 3D optimization loss function for unseen novel views of a given object [22, 30, 36].”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “Modern zero-shot text-to-image generators [66, 71, 65, 72, 2, 19] allowed to improve these results by providing stronger synthesis priors [62, 86, 52, 10, 12].”
Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “We also compare with the state-of-the-art approach of Latent-NeRF [23] under this user-guided generation setting; results are given in Fig.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “3.3)”
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “Initially, score distillation was adopted to harness the generation power of image diffusion models for 3D content (texture) synthesis [87, 51, 62, 70].”
LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation (2024, result): “The U-Net is built with residual layers [13] and self-attention layers [50] similar to previous works [14, 31, 46].”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, positive): “One such technique is the Latent-NeRF model, proposed in [113], which generates 3D shapes and textures from 2D”
LRM: Large Reconstruction Model for Single Image to 3D (2023, neutral): “…2022), to introduce semantics and multi-view guidance for image-to-3D reconstruction (Liu et al., 2023b; Tang et al., 2023; Deng et al., 2023; Shen et al., 2023b; Anciukeviˇcius et al., 2023; Melas-Kyriazi et al., 2023; Metzer et al., 2023; Xu et al., 2023; Qian et al., 2023; Li et al., 2023a).”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, methodology): “…models has enabled rapid progress in the domain of generating 3D content from scratch, either by optimizing radiance ﬁelds through vision-language models like CLIP [14, 16] or via text-conditioned diffusion models [35, 37, 36] as presented in DreamFusion [33] and its follow-ups [47, 19, 23].”
Adversarial Diffusion Distillation (2023, background): “While the majority of SDS-based works [45, 51, 68, 69] use SDS in the context of per-scene optimization for 3D objects, the approach has also been applied to text-to-3D-video-synthesis [62] and in the context of image editing [16].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, neutral): “Notably, models like DreamFusion [49], Magic3D [31], and Prolific-Dreamer [71] have pioneered a line of approach for per-shape optimization [6, 7, 12, 23, 29, 40, 43–45, 50, 52, 60, 63, 64, 67, 76, 77, 81].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “To enable the generation of 3D models from a single image, some works [43, 12, 46] utilize textual inversion [16], to ﬁnd the best-matching text embedding for the input image, which is then fed into a diffusion model.”