Improved Techniques for Training Consistency Models

By Yang Song, Prafulla Dhariwal. International Conference on Learning Representations, 2023

451 citations recorded by the source corpus.

Consistency models are a nascent family of generative models that can sample high quality data in one step without the need for adversarial training. Current consistency models achieve optimal sample quality by distilling from pre-trained diffusion models and employing learned metrics such as LPIPS. However, distillation limits the quality of consistency models to that of the pre-trained diffusion model, and LPIPS causes undesirable bias in evaluation. To tackle these challenges, we present improved techniques for consistency training, where consistency models learn directly from data without distillation. We delve into the theory behind consistency training and identify a previously overlooked flaw, which we address by eliminating Exponential Moving Average from the teacher consistency model. To replace learned metrics like LPIPS, we adopt Pseudo-Huber losses from robust statistics. Additionally, we introduce a lognormal noise schedule for the consistency training objective, and propose to double total discretization steps every set number of training iterations. Combined with better hyperparameter tuning, these modifications enable consistency models to achieve FID scores of 2.51 and 3.25 on CIFAR-10 and ImageNet $64\times 64$ respectively in a single sampling step. These scores mark a 3.5$\times$ and 4$\times$ improvement compared to prior consistency training approaches. Through two-step sampling, we further reduce FID scores to 2.24 and 2.77 on these two datasets, surpassing those obtained via distillation in both one-step and two-step settings, while narrowing the gap between consistency models and other state-of-the-art generative models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/b0d791cd8e9a87f37258832beaa321037048da26

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “Consistency Distillation [21, 32, 53, 74, 76] trains the student to map any points on an ODE trajectory to its origin.”
One Step Diffusion via Shortcut Models (2024, unknown): “Consistency models (Song et al., 2023) are a family of one-step generative models that learn to map from any partially-noised data point to the final data point.”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, result): “Consistency Distillation [9,12,19,26,31,32], and TRACT [33], train student models so their outputs are self-consistent at any timesteps along the ODE trajectory, and thus consistent with the teacher.”
Mean Flows for One-step Generative Modeling (2025, unknown): “While related to previous one-step generative models [46, 43, 14, 31, 49, 23, 12, 52], our method provides a more principled framework.”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, unknown): “Improved techniques for both Consistency Distillation [56, 33, 15, 68] and Progressive Distil-lation [35, 28, 3] have since been introduced.”