VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

By Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao-Liang Weng, Ying Shan. Computer Vision and Pattern Recognition, 2024

654 citations recorded by the source corpus.

Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with mini-mal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/492bc8339d8aac442c4ec13f8c1d59e980a3af2f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Zhiling Yan
2. Kepan Nan
3. Yue Huang
4. Yuan Li
5. Kai Zhang
6. Zhengqing Yuan
7. Yixin Liu
8. Chujie Gao

In Signal Processing

1. Zhenheng Yang
2. Zhijie Chen
3. Lifang He
4. Lichao Sun

Citations in the corpus

• Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Following recent advances [3, 5, 6, 15, 16, 32, 122, 126], we adopt the Two-Alternative Forced Choice protocol.”
• LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Other works (Guo et al., 2023; Zhang et al., 2024) focus on constructing the personalized text-to-video model to partially train temporal modules which can be adapted to various different personalized text-to-image models for personalized video generation.”
• OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods [19, 20, 7, 21, 22, 23, 24], and DiT based methods [6, 8, 5].”
• CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (2024, neutral): “…top-performing text-to-video models as baselines, including T2V-Turbo (Li et al., 2024), AnimateDiff (Guo et al., 2023), VideoCrafter2 (Chen et al., 2024a), OpenSora (Zheng et al., 2024b), Show-1 (Zhang et al., 2023a), Gen-2 (runway, 2023), Pika (pik, 2023), and LaVie-2 (Wang et al.,…”
• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “In addition, a text prompt can instruct the model to animate these elements by e.g., adding layers of movement, interaction, and narrative progression that bring the static image to life [27, 91, 92].”