Latte: Latent Diffusion Transformer for Video Generation
By Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, Yu Qiao. Trans. Mach. Learn. Res., 2024
570 citations recorded by the source corpus.
We propose Latte, a novel Latent Diffusion Transformer for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, i.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to the text-to-video generation (T2V) task, where Latte achieves results that are competitive with recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation.
View Paper: https://www.semanticscholar.org/paper/e0eac8c64be3313e581c28a495bec192e7e67284
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Yu Qiao
- Limin Wang
- Ziwei Liu
- Luc Van Gool
- Yinan He
- Xiaoou Tang
- Christoph Feichtenhofer
- Yuanhan Zhang
Citations in the corpus
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Unlike previous DiT architectures [3, 6] that focus on modeling the visual content, our MVDiT features a parallel visual-text architecture to enhance the consistency between the text and generated video.”
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, unknown): “Moreover, VDT (Lu et al., 2024) and Latte (Ma et al., 2024b) show DiTs can be extended for video generation through a space-time attention (Arnab et al., 2021).”
- CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, unknown): “…proposed, leveraging control signals such as optical flow (Yin et al., 2023; Chen et al., 2023b; Shi et al., 2024), pose skeleton (Ma et al., 2023; Ruiz et al., 2023), and other multi-modal signals (Wang et al., 2024; Ruan et al., 2023), enabling more accurate control for guiding video generation.”
- VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “We then elaborate on the prompt suites we use in Section 3.2.”
- Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “Latent Diffusion Transformer (Latte) [66] conducts extensive experiments among four model variants from the perspective of decomposing spatial and temporal information in video data.”
- Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, unknown): “Unlike previous methods (Ma et al., 2024) that utilize factorized spatial and temporal attention to reduce computational complexity, we directly employ full sequence attention, thanks to much fewer tokens required by our pyramidal representation.”