Zero-Shot Text-Guided Object Generation with Dream Fields

By Ajay Jain, Ben Mildenhall, Jonathan T. Barron, Pieter Abbeel, Ben Poole. Computer Vision and Pattern Recognition, 2021

672 citations recorded by the source corpus.

We combine neural rendering with multi-modal image and text representations to synthesize diverse 3D objects solely from natural language descriptions. Our method, Dream Fields, can generate the geometry and color of a wide range of objects without 3D supervision. Due to the scarcity of diverse, captioned 3D data, prior methods only generate objectsfrom a handful of categories, such as ShapeNet. Instead, we guide generation with image-text models pre-trained on large datasets of captioned images from the web. Our method optimizes a Neural Radiance Field from many camera views so that rendered images score highly with a target caption according to a pre-trained CLIP model. To improve fidelity and visual quality, we introduce simple geometric priors, including sparsity-inducing transmittance regularization, scene bounds, and new MLP architectures. In experiments, Dream Fields produce realistic, multi-view consistent object geometry and color from a variety of natural language captions.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/03e1c3b5fdad9b21bbed3d13af7e8d6c73cbcfa6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, methodology): “Earlier works employed CLIP [63] guidance for 3D generation [31, 29, 56, 94] and manipulation [60, 40, 21] from text prompts.”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “Some works (Jain et al., 2022; Mohammad Khalid et al., 2022) optimize meshes or NeRFs to minimize the CLIP Radford et al. (2021) score between the rendered images and input prompt utilizing pretrained CLIP model.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “To achieve open-vocabulary 3D generation, several methods propose to lift 2D image models for 3D generation (Jain et al., 2022; Poole et al., 2022; Wang et al., 2023a; Mohammad Khalid et al., 2022; Michel et al., 2022).”
DreamFusion: Text-to-3D using 2D Diffusion (2022, methodology): “We also compare against an enhanced reimplementation of Dream Fields where we use our own 3D representation (Sec.”
Text2LIVE: Text-Driven Layered Image and Video Editing (2022, methodology): “Several works [11,14,21,28] take a test-time optimization approach and leverage CLIP without using a pre-trained generator.”
Text-to-3D using Gaussian Splatting (2023, neutral): “, 2021), Dream Fields (Jain et al., 2022), Text2Mesh (Michel et al.”
Point-E: A System for Generating 3D Point Clouds from Complex Prompts (2022, methodology): “Inspired by these results, recent works have explored text-conditional generation in other modalities, such as video (Hong et al., 2022; Singer et al., 2022; Ho et al., 2022b;a) and 3D objects (Jain et al., 2021; Poole et al., 2022; Lin et al., 2022a; Sanghi et al., 2021; 2022).”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “…variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation [1,23,33,52,57,77,91].”
RealFusion 360° Reconstruction of Any Object from a Single Image (2023, background): “Most of these approaches train deep networks specifically for the goal of inferring NeRFs from a small number of views; examples include IBRNet [44], NeRF-WCE [9], Pixel-2”
MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers (2023, methodology): “Diffusion-based neural field synthesis in MLP weight spaces [13] and tri-planes [55] have also been explored, alongside leveraging image-based models for optimizing NeRFs [25, 34, 48, 63].”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Another branch of work employs CLIP to guide the text-conditioned generation of 3D objects [32, 41, 65, 83] or stylized meshes [45, 47] by encoding descriptive textual input.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “There are also models [18, 22, 29, 44, 46, 59, 75, 81] which adopt CLIP [56] to align each view of the 3D representation model with the text.”
FSGS: Real-Time Few-shot View Synthesis using Gaussian Splatting (2023, background): “Tremendous works focus on improving its efficiency [7, 15, 17, 26, 31, 38, 43], quality [1, 3, 4, 9, 20, 42, 48, 52], generalizing to unseen scenes [8, 10, 24, 44, 53, 59], applying artistic effects [14, 22, 49, 60] and 3D genera-2 tion [5, 6, 18, 21, 25, 27, 28, 34, 41, 45].”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “The score distillation sampling (SDS) proposed by DreamFusion [35] exhibits superior performance on zero-shot text-to-3D synthesis and outperforms CLIP-guided alternatives [15, 37, 57] significantly.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, methodology): “[32] use CLIP to edit or generate a single-object NeRF with a text prompt query by optimizing the NeRF parameters to generate images matched with the text.”
CLIP-Mesh: Generating textured meshes from text using pretrained image-text models (2022, methodology): “Comparison with [Jain et al. 2021] results from their paper/project website.”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “To address these data limitations, recent methods [56, 29, 41, 38, 88] lift the expressive power of 2D text-to-image models into 3D by formulating 3D generation as an iterative optimization problem in the image domain.”