LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models

By Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Pe-der Yang and 10 more. International Journal of Computer Vision, 2023

404 citations recorded by the source corpus.

This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously (a) accomplish the synthesis of visually realistic and temporally coherent videos while (b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: (1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. (2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications. Project page: https://github.com/Vchitect/LaVie/.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ed4603ea341acc26cab24f41aa40524fb7779917

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yu Qiao
3. Chen Change Loy
4. Xiaogang Wang
5. Ming-Hsuan Yang
6. Dacheng Tao
7. Ying Shan
8. Ziwei Liu

Citations in the corpus

• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “Similar to Imagen Video, LAVIE [62] proposes to transform a T2I model into a T2V model with cascaded LDMs, with two key insights: i) using simply temporal self-attention and RoPE [63] can adequately capture temporal correlations inherent in video data; ii) joint image-video fine-tuning enable more…”
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “Similar methods are also been proposed by SHOW-1 [65] and LAVIE [59] T2V generation also raises the enthusiasm of commerce or non-commerce companies.”
• VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “To leverage the prior in SD trained on a large-scale image dataset, most text-to-video diffusion models inflate the SD model to a video model by adding temporal modules, including Align Your Latent [14], AnimateDiff [22], LVDM [23], Magic Video [63], Mod-elScopeT2V [48], and LAVIE [50].”
• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, methodology): “Recently, diffusion models [19, 38, 89, 91] have achieved significant progress in image synthesis [31, 46, 47, 78, 81, 84, 86], and enabled a line of works towards video generation [8, 13, 28, 33– 35, 39, 40, 50, 58, 72, 88, 98, 104, 118, 124, 132, 133].”
• AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning (2023, background): “Since the setting in this report is newly proposed, there is currently little work targeting it.”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Extra refinements on the prediction results [6, 49, 54, 99, 126, 144] could be helpful.”
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, neutral): “…top-performing text-to-video models as baselines, including T2V-Turbo (Li et al., 2024), AnimateDiff (Guo et al., 2023), VideoCrafter2 (Chen et al., 2024a), OpenSora (Zheng et al., 2024b), Show-1 (Zhang et al., 2023a), Gen-2 (runway, 2023), Pika (pik, 2023), and LaVie-2 (Wang et al., 2023b).”
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “Another commonality between these and most subsequent approaches to (text-to-)video synthesis [29, 97, 99] is the usage of dedicated expert models to generate the actual visual content at a coarse frame rate and to temporally upscale this low-fps video to temporally smooth final outputs at 24-32…”
• Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, positive): “Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020) are generative models that utilize a reverse Markov chain to synthesize data, beginning from a noise distribution and progressively denoising it.”
• I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, background): “Imagen Video [13] and Lavie [47] synthesize high-definition videos to achieve performance improvement in a progressive manner.”
• OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “For example, based on DiT [2], OpenSora 2 , OpenSoraPlan [3] and recent works [4, 5] utilize the collected million-scale text-video datasets to reproduce Sora.”
• WorldSimBench: Towards Video Generation Models as World Simulators (2024, neutral): “We evaluate 8 popular video generation model, including Open-Sora-Plan(T2V) (Lab & etc., 2024), Lavie (Wang et al., 2023c), ModelScope (Wang et al., 2023b), OpenSora (Zheng et al., 2024), Ani-mateDiff (Guo et al., 2023), Open-Sora-Plan(TI2V) (Lab & etc., 2024), DynamicCrafter (Xing et al., 2023),…”