Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models

By Jingfeng Yao, Xinggang Wang. Computer Vision and Pattern Recognition, 2025

348 citations recorded by the source corpus.

Latent diffusion models with Transformer architectures excel at generating high-fidelity images. However, recent studies reveal an optimization dilemma in this two-stage design: while increasing the per-token feature dimension in visual tokenizers improves reconstruction quality, it requires substantially larger diffusion models and more training iterations to achieve comparable generation performance. Consequently, existing systems often settle for suboptimal solutions, either producing visual artifacts due to information loss within tokenizers or failing to converge fully due to expensive computation costs. We argue that this dilemma stems from the inherent difficulty in learning unconstrained high-dimensional latent spaces. To address this, we propose aligning the latent space with pretrained vision foundation models when training the visual tokenizers. Our proposed VA-VAE (Vision foundation model Aligned Variational AutoEncoder) significantly expands the reconstruction-generation frontier of latent diffusion models, enabling faster convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces. To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with improved training strategies and architecture designs, termed LightningDiT. The integrated system achieves state-of-the-art (SOTA) performance on ImageNet 256×256 generation with an FID score of 1.35 while demonstrating remarkable training efficiency by reaching an FID score of 2.11 in just 64 epochs – representing an over 21× convergence speedup compared to the original DiT. Models and codes are available at https://github.com/hustvl/LightningDiT.

Subfields: Computer Networks and Communications, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/4e413bf865ec893338000817a15d36f6f136ec8f

Citations in the corpus