Pyramidal Flow Matching for Efficient Video Generative Modeling
By Yang Jin, Zhicheng Sun, Ning Li, Kun Xu, Hao Jiang, Zhuang Nan, Quzhe Huang, Yang Song, Yadong Mu, Zhouchen Lin. International Conference on Learning Representations, 2024
347 citations recorded by the source corpus.
Video generation requires modeling a vast spatiotemporal space, which demands significant computational resources and data usage. To reduce the complexity, the prevailing approaches employ a cascaded architecture to avoid direct training with full resolution latent. Despite reducing computational demands, the separate optimization of each sub-stage hinders knowledge sharing and sacrifices flexibility. This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours. All code and models are open-sourced at https://pyramid-flow.github.io.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/457d4cffbb11666180507db21a86435941a69625
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Eli Shechtman
- Xun Huang
- Mingyuan Zhou
- Zhengqi Li
- Guande He
- Luc Van Gool
- Ming-Hsuan Yang
- Trevor Darrell
Citations in the corpus
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “For Teacher Forcing, we describe an efficient variant that processes all frames in parallel using block sparse attention masks, rather than denoising one frame at each training iteration [33].”
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, neutral): “We apply causal attention masks to the model and fine-tune it with the autoregressive training method described in Sec.”