Diffusion models as plug-and-play priors

By Alexandros Graikos, Esmeralda S. Whitammer, Nebojsa Jojic, Dimitris Samaras. Neural Information Processing Systems, 2022

309 citations recorded by the source corpus.

We consider the problem of inferring high-dimensional data $\mathbf{x}$ in a model that consists of a prior $p(\mathbf{x})$ and an auxiliary differentiable constraint $c(\mathbf{x},\mathbf{y})$ on $x$ given some additional information $\mathbf{y}$. In this paper, the prior is an independently trained denoising diffusion generative model. The auxiliary constraint is expected to have a differentiable form, but can come from diverse sources. The possibility of such inference turns diffusion models into plug-and-play modules, thereby allowing a range of potential applications in adapting models to new domains and tasks, such as conditional generation or image segmentation. The structure of diffusion models allows us to perform approximate inference by iterating differentiation through the fixed denoising network enriched with different amounts of noise at each step. Considering many noised versions of $\mathbf{x}$ in evaluation of its fitness is a novel search mechanism that may lead to new algorithms for solving combinatorial optimization problems.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/44b3fb61bec49f5e4d8a3b81ddc6082aedd771fe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jun Zhu
2. Hang Su
3. Ming-Hsuan Yang
4. Chongxuan Li
5. Ping Luo
6. Tom Goldstein
7. Micah Goldblum
8. Fan Bao

Citations in the corpus

• Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation (2022, methodology): “Differently, their procedure uses the mathematical setup by Graikos et al. [12] to search for image parametrization that minimizes the training loss of a diffusion model.”
• Universal Guidance for Diffusion Models (2023, background): “Prior work that studied guided image generation did so with a variety of restrictions and external guidance functions [3, 4, 6, 7, 12, 16, 28].”
• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “Generative pre-training can enhance the label utilization of semantic segmentation models, and recent work has shown that representations learned through DDPM contain high-level semantic information that is useful for segmentation tasks [9, 76].”
• LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching (2023, methodology): “Hence, ISM aligns with the fundamental principles of SDS-like objectives [9, 33, 45] albeit in a more refined manner.”
• Diffusion Self-Guidance for Controllable Image Generation (2023, background): “Generative image models have improved rapidly in recent years with the adoption of large text-image datasets and scalable architectures [7, 10, 12, 27, 28, 32, 34, 38].”
• DiffusionDet: Diffusion Model for Object Detection (2022, methodology): “Some pioneer works tried to adopt the diffusion model for image segmentation tasks [1, 5, 6, 13, 31, 47, 97], for example, Chen et al .”
• ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation (2023, result): “In addition, adjusting time schedule has also been discussed in previous works [12, 55].”