DreamFusion: Text-to-3D using 2D Diffusion
By Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall. 2022
3,724 citations recorded by the source corpus.
Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.
Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Graphics and Computer-Aided Design
- Jonathan T. Barron
- Andreas Geiger
- Ben Mildenhall
- Pratul P. Srinivasan
- Matthias Nießner
- Peng Wang
- Andrea Tagliasacchi
- Peter Hedman
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrea Vedaldi
- Andrew Zisserman
In Generative Models
- Eli Shechtman
- Richard Zhang
- Daniel Cohen-Or
- Jun-Yan Zhu
- Alexei A. Efros
- Ming-Hsuan Yang
- Arash Vahdat
- Robin Rombach
In Multimodal Learning
- Daniel Cohen-Or
- Alex Nichol
- Alexei A. Efros
- Hongsheng Li
- Ying Shan
- Heewoo Jun
- Ziwei Liu
- Ludwig Schmidt
Citations in the corpus
- Foundation Models and Fair Use (2023, background): “There are already mechanisms to generate 3D models from text (Poole et al., 2022).”
- Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior (2023, methodology): “Notably, DreamFusion [32] uses a textto-image diffusion model to guide the optimization of the 3D representation.”
- One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “We compare our method with Stable Dreamfusion [55] and 3DFuse [68].”
- Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures (2022, methodology): “[35].”
- Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, neutral): “Diffusion Models: Diffusion models [35, 69] have recently gained significant attention from the research community due to their ability to generate high-fidelity and diverse content like images [66, 54, 24], videos [68, 34, 77], 3D [58, 49], and audio [43, 51] from various input modalities like text.”
- Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, methodology): “To extend its application scenarios, numerous efforts [1,23, 92,95] have been made to incorporate 3D point clouds with data from other modalities, allowing for improved 3D understanding [1, 23], text-to-3D generation [35, 49, 52], and 3D question answering [3, 28].”
- InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “The score distillation sampling (SDS) proposed by DreamFusion [35] exhibits superior performance on zero-shot text-to-3D synthesis and outperforms CLIP-guided alternatives [15, 37, 57] significantly.”
- LRM: Large Reconstruction Model for Single Image to 3D (2023, positive): “…descriptions for 3D shapes by applying BLIP (Li et al., 2023b) to generate captions of different views, uses GPT-4 (OpenAI, 2023) to summarize them, and then employs these language-3D pairs for training text-to-3D generative models (Nichol et al., 2022; Poole et al., 2022; Jun & Nichol, 2023).”
- Improved Distribution Matching Distillation for Fast Image Synthesis (2024, neutral): “This training objective proved more stable than GAN-based methods and has demonstrated superior performance in one-step image synthesis.”
- Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “A growing body of research [5, 31, 47, 63, 67], such as DreamField [24], DreamFusion [47], and Magic3D [31], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sampling (SDS) [47].”
- HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “…variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation [1,23,33,52,57,77,91].”
- Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation (2023, methodology): “We consistently outperform DreamFusion in both appearance and geometry generation.”
- Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “Influences by DreamFusion.”
- PGSR: Planar-Based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction (2024, neutral): “N OVEL view synthesis and geometry reconstruction are challenging and crucial tasks in computer vision, widely used in AR/VR [1], [2], 3D content generation [3], [4], [5], [6], and autonomous driving.”
- Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Photo-realistic and real-time rendering of 3D scenes has always been a pivotal interest in both academic research and industrial domains, with applications spanning virtual reality [51], media generation [36], and large-scale scene visualization [43, 45, 49].”
- SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “However, they are prone to artifacts like Janus problem [30, 36] and over-saturated texture.”
- CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets (2024, neutral): “We have presented CLAY, a large-scale 3D generative model that supports multi-modal controls for high quality 3D asset generation, further bridging the gap between the vivid realms of human imagination and the tangible world of digital creation.”