Perception Prioritized Training of Diffusion Models

By Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo J. Kim, Sung-Hoon Yoon. 2022

401 citations recorded by the source corpus.

Diffusion models learn to restore noisy data, which is corrupted with different levels of noise, by optimizing the weighted sum of the corresponding loss terms, i.e., denoising score matching loss. In this paper, we show that restoring data corrupted with certain noise levels offers a proper pretext task for the model to learn rich visual concepts. We propose to prioritize such noise levels over other levels during training, by redesigning the weighting scheme of the objective function. We show that our simple redesign of the weighting scheme significantly improves the performance of diffusion models regardless of the datasets, architectures, and sampling strategies.

Subfields: Computer Vision and Pattern Recognition, Generative Models, Optimization & Training, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

In Optimization & Training

In Signal Processing

Citations in the corpus

Elucidating the Design Space of Diffusion-Based Generative Models (2022, background): “In concurrent work, Choi et al. [6] propose a similar scheme to prioritize noise levels that are most relevant w.r.t. forming the perceptually recognizable content of the image.”
Diffusion Models already have a Semantic Latent Space (2022, methodology): “, 2020) on ADM with P2-weighting (Dhariwal & Nichol, 2021) (Choi et al., 2022).”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “In addition, they apply a morphology levels prioritization module [148] that designates higher weight values to the loss at earlier levels to emphasize perceptual information and lower weights to the loss at later levels, resulting in higher fidelity samples.”
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment (2024, background): “As observed by [5,16,24], diffusion models typically predict low-frequency content during the initial stages of the denoising process and subsequently concentrate on high-frequency details towards the final stage.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, neutral): “…and Ho, 2022] (Appendix D.2) e − λ/ 2 Flow Matching with OT path (FM-OT) [Lipman et al., 2022] (Appendix D.3) e − λ/ 2 InDI [Delbracio and Milanfar, 2023] (Appendix D.4) e − λ sech 2 ( λ/ 4) P2 weighting with ‘cosine’ schedule [Choi et al., 2022] (Appendix D.5) sech ( λ/ 2) / (1 + e λ ) γ , γ = 0 .”
MACE: Mass Concept Erasure in Diffusion Models (2024, background): “Typically, diffusion models generate a general context in the early stage [12, 31, 51].”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “To further substantiate this, we employ the cosine similarity between the features of Stable Diffusion before and after the SFT to measure the condition strength provided by the encoder.”