Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation

By Dongjun Kim, Seung-Jae Shin, Kyungwoo Song, Wanmo Kang, Il-Chul Moon. 2021

118 citations recorded by the source corpus.

Recent advances in diffusion models bring state-of-the-art performance on image generation tasks. However, empirical results from previous research in diffusion models imply an inverse correlation between density estimation and sample generation performances. This paper investigates with sufficient empirical evidence that such inverse correlation happens because density estimation is significantly contributed by small diffusion time, whereas sample generation mainly depends on large diffusion time. However, training a score network well across the entire diffusion time is demanding because the loss scale is significantly imbalanced at each diffusion time. For successful training, therefore, we introduce Soft Truncation, a universally applicable training technique for diffusion models, that softens the fixed and static truncation hyperparameter into a random variable. In experiments, Soft Truncation achieves state-of-the-art performance on CIFAR-10, CelebA, CelebA-HQ 256x256, and STL-10 datasets.

Subfields: Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shuicheng Yan
2. Ming-Hsuan Yang
3. Luc Van Gool
4. Dacheng Tao
5. Pan Zhou
6. Xiaogang Wang
7. Yi Yang
8. Jitendra Malik

In Radiology, Nuclear Medicine and Imaging

1. Dongjun Kim
2. Shuicheng Yan
3. Mingg-Ming Cheng
4. Ming-Hsuan Yang
5. Pan Zhou
6. Stefano Ermon
7. Shanghua Gao

Citations in the corpus

• Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “Due to the flexibility of diffusion models, many empirical methods can be strengthened through further analysis [9, 99, 103, 189].”
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion (2023, background): “The selection of c skip and c out ensures that the initialized g θ ( x t , t, t ) closely ) (Song et al., 2021; Kim et al., 2022b), establishing a new SOTA NLL.”
• Building Normalizing Flows with Stochastic Interpolants (2022, unknown): “, 2019; Kingma & Dhariwal, 2018), as well as to recent advances in score based diffusion in DDPM, DDPM++, VDM, Score SDE, ScoreFlow, and Soft Truncation (Ho et al., 2020; Nichol & Dhariwal, 2021; Kingma et al., 2021; Song et al., 2021b;a; Kim et al., 2022).”
• Masked Diffusion Transformer is a Strong Image Synthesizer (2023, background): “Additionally, some training schemes [2,12] are introduced to improve the diffusion model training, e.g ., approximate maximum likelihood training [28,34,47], and training loss weighting [26,27].”