Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference

By Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, Hang Zhao. arXiv.org, 2023

876 citations recorded by the source corpus.

Latent Diffusion models (LDMs) have achieved remarkable results in synthesizing high-resolution images. However, the iterative sampling process is computationally intensive and leads to slow generation. Inspired by Consistency Models (song et al.), we propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs, including Stable Diffusion (rombach et al). Viewing the guided reverse diffusion process as solving an augmented probability flow ODE (PF-ODE), LCMs are designed to directly predict the solution of such ODE in latent space, mitigating the need for numerous iterations and allowing rapid, high-fidelity sampling. Efficiently distilled from pre-trained classifier-free guided diffusion models, a high-quality 768 x 768 2~4-step LCM takes only 32 A100 GPU hours for training. Furthermore, we introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets. Evaluation on the LAION-5B-Aesthetics dataset demonstrates that LCMs achieve state-of-the-art text-to-image generation performance with few-step inference. Project Page: https://latent-consistency-models.github.io/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8b7cce220c3b19f9b2d4a6c531907ed3b592b55e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image (2024, background): “The latent consistency models [38] may be leveraged to train a few-step diffusion model so that the inference time may be decreased to less than 1 second.”
InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (2023, methodology): “Inference Time FID-5k CLIP SD 1.4-DPM Solver (25 step)[5, 19] 0”
One Step Diffusion via Shortcut Models (2024, positive): “A number of prior works have explored the distillation of pre-trained diffusion models into a one-step or few-step model (Luo, 2023).”
Improved Distribution Matching Distillation for Fast Image Synthesis (2024, methodology): “Consistency Distillation [9,12,19,26,31,32], and TRACT [33], train student models so their outputs are self-consistent at any timesteps along the ODE trajectory, and thus consistent with the teacher.”
Exploiting Diffusion Prior for Real-World Image Super-Resolution (2023, neutral): “Fast sampling strategy and model distillation are two promising solutions to improve efﬁciency, and such exploration is left as our future work.”
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior (2023, background): “There’re works [33, 44] that can already achieve satisfactory generation performance with only 1 ∼ 4 steps, thus the time-consuming problem can be solved in the future.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Potential solutions may include faster sampling techniques [89, 146] and training-based distillations [91, 93, 101, 103, 110, 124].”
Adversarial Diffusion Distillation (2023, methodology): “• ADD signiﬁcantly outperforms strong baselines such as LCM, LCM-XL [38] and single-step GANs [59], and is able to handle complex image compositions while maintaining high image realism at only a single inference step.”
One-Step Diffusion with Distribution Matching Distillation (2023, methodology): “To accelerate sampling speed, previous methods [42, 43, 47, 48, 51, 65, 75, 90, 91] distill the noise image mapping, discovered by the original multi-step diffusion sampling, into a single-pass student network.”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, positive): “Consistency Distillation [21, 32, 53, 74, 76] trains the student to map any points on an ODE trajectory to its origin.”
Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding (2024, methodology): “Algorithmic Acceleration Recently, various methods have been proposed to reduce the inference steps of diffusion-based text-to-image models [22, 31, 21, 37, 40].”
Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation (2024, positive): “Improved techniques for both Consistency Distillation [56, 33, 15, 68] and Progressive Distil-lation [35, 28, 3] have since been introduced.”
MACE: Mass Concept Erasure in Diffusion Models (2024, methodology): “From this database, we sample 200 artists and split them into two groups: an erasure group of 100 artists and a retention group with 100 other artists.”
Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “To resolve these challenges, we employed various techniques to reduce inference cost and latency [Dao et al., 2022, Ainslie et al., 2023, Luo et al., 2023, Lin et al., 2023].”