DreamFusion: Text-to-3D using 2D Diffusion

By Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall. 2022

3,724 citations recorded by the source corpus.

Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image generator. Using this loss in a DeepDream-like procedure, we optimize a randomly-initialized 3D model (a Neural Radiance Field, or NeRF) via gradient descent such that its 2D renderings from random angles achieve a low loss. The resulting 3D model of the given text can be viewed from any angle, relit by arbitrary illumination, or composited into any 3D environment. Our approach requires no 3D training data and no modifications to the image diffusion model, demonstrating the effectiveness of pretrained image diffusion models as priors.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

  1. Jonathan T. Barron
  2. Andreas Geiger
  3. Ben Mildenhall
  4. Pratul P. Srinivasan
  5. Matthias Nießner
  6. Peng Wang
  7. Andrea Tagliasacchi
  8. Peter Hedman

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Ming-Hsuan Yang
  4. Xiaogang Wang
  5. Trevor Darrell
  6. Jitendra Malik
  7. Andrea Vedaldi
  8. Andrew Zisserman

In Generative Models

  1. Eli Shechtman
  2. Richard Zhang
  3. Daniel Cohen-Or
  4. Jun-Yan Zhu
  5. Alexei A. Efros
  6. Ming-Hsuan Yang
  7. Arash Vahdat
  8. Robin Rombach

In Multimodal Learning

  1. Daniel Cohen-Or
  2. Alex Nichol
  3. Alexei A. Efros
  4. Hongsheng Li
  5. Ying Shan
  6. Heewoo Jun
  7. Ziwei Liu
  8. Ludwig Schmidt

Citations in the corpus