ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

By Zhengyi Wang, Cheng Lu, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu. Neural Information Processing Systems, 2023

1,425 citations recorded by the source corpus.

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., $7.5$). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., $512\times512$) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page and codes: https://ml.cs.tsinghua.edu.cn/prolificdreamer/

Subfields: Biophysics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c5e9fd131cde68c218d0ea69cd617a67c7f35d42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Recent advances in 3D Gaussian splatting (2024, neutral): “It is promising to extract geometry, texture, and lighting with more advanced rendering techniques to facilitate independent editing and build the connection between 3DGS and mesh-based representation to enable efficient editing.”
SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution (2023, neutral): “It has been demonstrated that T2I diffusion priors are powerful in image editing [42, 72], video generation [50, 62], 3D content generation [37, 60, 60], etc .”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, background): “Research in this space has aimed to improve distillation strategies [18, 19, 20, 21, 22], swap in other 3D representations [23, 24, 25, 26, 27], and amortize the optimization process [28].”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, neutral): “…[54] and Stable Diffusion [56]) with impressive and high-quality image generation, a large number of studies [7, 33, 40, 50] incorporate 2D diffusion and CLIP priors into per-shape optimization via various score distillation sampling strategy [50, 62, 66], or with other monocular prior losses.”
Adversarial Diffusion Distillation (2023, background): “While the majority of SDS-based works [45, 51, 68, 69] use SDS in the context of per-scene optimization for 3D objects, the approach has also been applied to text-to-3D-video-synthesis [62] and in the context of image editing [16].”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, background): “Recent works have extended score distillation [36, 37, 40–42] to diffusion distillation [22, 43–45].”
One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (2023, methodology): “Baselines: We compared One-2-3-45++ with optimization-based methods, specifically ProlificDreamer [71] and MV-Dream [61], as well as a feed-forward approach, Shap-E [25].”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, background): “Lifting 2D diffusion models for 3D generation Given the limited amount of 3D data available for training, recent works have attempted to leverage 2D diffusion models to generate 3D assets from a text prompt [28, 37, 59, 61] or an input image [29, 48].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “A.6 A DDITIONAL D ISCUSSIONS ON P ROMPT R EWRITING S TRATEGY Very recently, OpenAI has released DELLE-3 (Betker et al., 2023), which proposes to improve generated image quality by rewriting descriptive and better prompts with GPT-4.”
CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “Other models such as SDFusion [Cheng et al. 2023] and ShapeGPT [Yin et al. 2023] adopt an intuitive 3D VAE (Variational Autoencoder) for encoding geometries and reconstructing SDF fields.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “…proposes Score Distillation Sampling (SDS) to address the 3D data limitation by distilling 3D geometry and appearance from powerful 2D diffusion models (Saharia et al., 2022), which inspires the development of recent 2D lifting methods (Lin et al., 2023; Wang et al., 2023b; Chen et al., 2023c).”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “However, SDS-based methods [3, 20, 49, 52] frequently encounter the multi-face issue, also known as the “Janus” problem.”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, background): “Subsequent studies have focused on enhancing 3D representations [6, 17, 42, 43], refining sampling schedules [13], and optimizing loss designs [46].”