Latent Video Diffusion Models for High-Fidelity Long Video Generation

By Yin-Yin He, Tianyu Yang, Yong Zhang, Ying Shan, Qifeng Chen. 2022

429 citations recorded by the source corpus.

AI-generated content has attracted lots of attention recently, but photo-realistic video synthesis is still challenging. Although many attempts using GANs and autoregressive models have been made in this area, the visual quality and length of generated videos are far from satisfactory. Diffusion models have shown remarkable results recently but require significant computational resources. To address this, we introduce lightweight video diffusion models by leveraging a low-dimensional 3D latent space, significantly outperforming previous pixel-space video diffusion models under a limited computational budget. In addition, we propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. To further overcome the performance degradation issue for long video generation, we propose conditional latent perturbation and unconditional guidance that effectively mitigate the accumulated errors during the extension of video length. Extensive experiments on small domain datasets of different categories suggest that our framework generates more realistic and longer videos than previous strong baselines. We additionally provide an extension to large-scale text-to-video generation to demonstrate the superiority of our work. Our code and models will be made publicly available.

View Paper: https://www.semanticscholar.org/paper/cb4b91a6a7103410c1bcde9df69ab641c2e79a10

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Lijun Yu
2. D. Kondratyuk
3. Andreas Blattmann
4. Haoxin Chen
5. José Lezama
6. Agrim Gupta
7. Lu Jiang
8. Yong Zhang

Citations in the corpus

• VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Most video generation works use diffusion-based methods for text-to-video [4, 5, 25, 30, 33, 56, 67, 69, 70, 80, 81, 81, 85] and video-to-video editing [15, 23, 24, 39].”
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, background): “Latent Video Diffusion Models Video-LDMs [9, 31, 32, 35, 97] train the main generative model in a latent space of reduced computational complexity [22, 71].”
• Latte: Latent Diffusion Transformer for Video Generation (2024, unknown): “…recent methods to quantitatively evaluate the outcomes, including MoCoGAN (Tulyakov et al, 2018), VideoGPT (Yan et al, 2021), MoCoGAN-HD (Tian et al, 2021), DIGAN (Yu et al, 2022), StyleGAN-V (Sko-rokhodov et al, 2022), PVDM (Yu et al, 2023), MoStGAN-V (Shen et al, 2023) and LVDM (He et al, 2023).”
• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Swayed by the success of diffusion models [24, 48, 100], a surge of diffusion-based video generation methods have emerged [6, 34, 46, 51, 108, 117, 121].”
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “ModelScope-XL [57] ZeroScope [5] Floor33 [20] PikaLab [4] Gen2 [16] VideoQuality follows: Face Consistency.”
• MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, unknown): “The structure of the supplementary materials is as follows: • More results of MotionCtrl based on LVDM [9].”
• Photorealistic Video Generation with Diffusion Models (2023, background): “Video diffusion models can be categorized into pixel-space [34, 36, 66] and latent-space [4, 24, 31, 83] approaches, the later bringing important efficiency advantages when modeling videos.”