Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

By Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, Gregory Shakhnarovich. Computer Vision and Pattern Recognition, 2022

701 citations recorded by the source corpus.

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and re-purposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION 5B dataset.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fc011ed5ee986332523a62d2783adee1179dc1ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Wonder3D: Single Image to 3D Using Cross-Domain Diffusion (2023, methodology): “A growing body of research [5, 31, 47, 63, 67], such as DreamField [24], DreamFusion [47], and Magic3D [31], resort to distilling prior knowledge of 2D image diffusion models or vision language models to create 3D models from text or images via Score Distillation Sampling (SDS) [47].”
DreamLLM: Synergistic Multimodal Comprehension and Creation (2023, background): “…in 2D, a series of works have significantly propelled the 3D synthesis development (Mildenhall et al., 2022; Liu et al., 2023d; Lin et al., 2023; Wang et al., 2023c; Tang et al., 2023) based on Score Distillation Sampling (SDS) (Poole et al., 2023; Wang et al., 2023a) that utilizes pretrained…”
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model (2023, methodology): “To address this, another line of methods (Poole et al., 2022; Wang et al., 2023a; Lin et al., 2023; Wang et al., 2023b; Chen et al., 2023c) leverage the semantic understanding and high-quality generation capabilities of pretrained 2D diffusion models.”
ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, methodology): “There are three main differences between our proposed VSD and previous SDS [34] / SJC [55]: (1) VSD can optimize multiple ( n ≥ 1 ) 3D objects, while SDS / SJC only optimizes a single ( n = 1 ) 3D object.”
GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (2023, methodology): “Here come two main streams for achieving this goal: (i) training a new diffusion model with 3D data [12, 15, 24, 48] (namely the 3D diffusion model ) and (ii) lifting the 2D diffusion model to 3D [2, 6, 9, 32, 52, 58, 67, 70, 76, 77, 81, 90].”
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers (2023, positive): “…[54] and Stable Diffusion [56]) with impressive and high-quality image generation, a large number of studies [7, 33, 40, 50] incorporate 2D diffusion and CLIP priors into per-shape optimization via various score distillation sampling strategy [50, 62, 66], or with other monocular prior losses.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, methodology): “Such a trend can also be found on RTMV in Figure 6, even though the scenes are out-of-distribution compared to the Objaverse DietNeRF [23] Image Variation [1] SJC-I [53] Ours”
Latte: Latent Diffusion Transformer for Video Generation (2024, neutral): “2023; Zhang et al, 2023), and 3D object generation (Wang et al, 2023a; Chen et al, 2023b; Zhou et al, 2021; Shue et al, 2023), etc. Compared to these successful applications in images, generating high-quality videos still faces significant challenges, which can be primarily attributed to the…”
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors (2023, background): “Modern zero-shot text-to-image generators [66, 71, 65, 72, 2, 19] allowed to improve these results by providing stronger synthesis priors [62, 86, 52, 10, 12].”
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (2023, methodology): “To address these data limitations, recent methods [56, 29, 41, 38, 88] lift the expressive power of 2D text-to-image models into 3D by formulating 3D generation as an iterative optimization problem in the image domain.”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, methodology): “This also helps avoid certain design decisions that result in the cartoon-ish appearance commonly seen in unconditional 3D content generation methods [33, 47, 19].”
ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation (2023, methodology): “Alongside Dreamfusion, works like SJC[44], which utilize stable-diffusion models [33], have emerged.”
Shap-E: Generating Conditional 3D Implicit Functions (2023, methodology): “Wang et al. [65] propose a different approach for leveraging text-to-image diffusion models, using them to optimize a differentiable 3D voxel grid rather than an MLP-based NeRF.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, methodology): “Therefore, recent methods [8, 38, 49, 73, 77] resort to distilling pretrained text-to-image diffusion models for creating 3D models from texts, which shows impressive results on this text-to-3D task.”
InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models (2024, methodology): “However, SDS-based methods [3, 20, 49, 52] frequently encounter the multi-face issue, also known as the “Janus” problem.”
Objaverse-XL: A Universe of 10M+ 3D Objects (2023, methodology): “As shown in [15], the pretrained view-conditioned diffusion model can also be plugged into a score distillation framework such as DreamFusion [47] or SJC [54] to obtain a 3D assets.”
MVDream: Multi-view Diffusion for 3D Generation (2023, methodology): “Concurrent with Dreamfusion, SJC (Wang et al., 2023a) proposed a similar technique using the publicly available stable-diffusion model (Rombach et al., 2022).”
ReconFusion: 3D Reconstruction with Diffusion Priors (2023, positive): “Lifting 2D diffusion models for 3D generation Given the limited amount of 3D data available for training, recent works have attempted to leverage 2D diffusion models to generate 3D assets from a text prompt [28, 37, 59, 61] or an input image [29, 48].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “The images are then fed to the CLIP model [22, 25, 47, 34, 3, 31, 2, 27, 83, 38] or 2D diffusion model [55, 35, 68, 43, 12, 72, 82, 46, 93, 57] for calculating the loss functions, which are used to guide the 3D shape optimization.”