Estimating the Optimal Covariance with Imperfect Mean in Diffusion Probabilistic Models

By Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu. 2022

91 citations recorded by the source corpus.

Diffusion probabilistic models (DPMs) are a class of powerful deep generative models (DGMs). Despite their success, the iterative generation process over the full timesteps is much less efficient than other DGMs such as GANs. Thus, the generation performance on a subset of timesteps is crucial, which is greatly influenced by the covariance design in DPMs. In this work, we consider diagonal and full covariances to improve the expressive power of DPMs. We derive the optimal result for such covariances, and then correct it when the mean of DPMs is imperfect. Both the optimal and the corrected ones can be decomposed into terms of conditional expectations over functions of noise. Building upon it, we propose to estimate the optimal covariance and its correction given imperfect mean by learning these conditional expectations. Our method can be applied to DPMs with both discrete and continuous timesteps. We consider the diagonal covariance in our implementation for computational efficiency. For an efficient practical implementation, we adopt a parameter sharing scheme and a two-stage training process. Empirically, our method outperforms a wide variety of covariance design on likelihood results, and improves the sample quality especially on a small number of timesteps.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps (2022, background): “Samplers that need extra training or optimization include knowledge distillation [13, 14], learning the noise level or variance [15, 16, 33], and learning the noise schedule or sample trajectory [17, 18].”
eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers (2022, methodology): “Samples are drawn from these diffusion models using classifier(-free) guidance [13, 28] as well as various sampling algorithms that use deterministic [15, 34, 43, 46, 70, 89] or stochastic [4, 5, 16, 90] iterative updates.”
DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models (2022, unknown): “Samplers based on extra training include model distillation (Salimans & Ho, 2022; Luhman & Luhman, 2021), learning reverse process variances (San-Roman et al., 2021; Nichol & Dhariwal, 2021; Bao et al., 2022a), and learning sampling steps (Lam et al., 2021; Watson et al., 2022).”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “These models are trained via denoising autoencoder objectives that can be interpreted as score matching [84], and generate samples via an iterative denoising procedure that may use stochastic updates [5, 6, 15, 98] which solve stochastic differential equations (SDEs) or deterministic updates [14,30,”
A Survey on Generative Diffusion Models (2022, methodology): “Analytic-DPM [82] and extended Analytic-DPM [83] jointly propose optimal reverse solutions under correction for each state.”
Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Additionally, some training schemes [2,12] are introduced to improve the diffusion model training, e.g ., approximate maximum likelihood training [28,34,47], and training loss weighting [26,27].”
UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models (2023, background): “[2] Fan Bao, Chongxuan Li, Jiacheng Sun, Jun Zhu, and Bo Zhang.”