VideoCrafter1: Open Diffusion Models for High-Quality Video Generation

By Haoxin Chen, Menghan Xia, Yin-Yin He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang and 2 more. arXiv.org, 2023

596 citations recorded by the source corpus.

Video generation has increasingly gained interest in both academia and industry. Although commercial tools can generate plausible videos, there is a limited number of open-source models available for researchers and engineers. In this work, we introduce two diffusion models for high-quality video generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V models synthesize a video based on a given text input, while I2V models incorporate an additional image input. Our proposed T2V model can generate realistic and cinematic-quality videos with a resolution of $1024 \times 576$, outperforming other open-source T2V models in terms of quality. The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style. This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints. We believe that these open-source video generation models will contribute significantly to the technological advancements within the community.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1891c3756f870d902a0b793a1dcd5cc34c778612

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “As a concurrent method, Videocrafter1 [4, 50] leverages a diffusion model to address high-quality video generation tasks and places significant emphasis on fostering the growth of the community.”
VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models (2024, methodology): “We compare our approach with several state-of-the-art T2V models, including popular commercial models such as Gen-2 [5] and Pika Labs [9], as well as open-source models like Show-1 [60], VideoCrafter1 [15], and AnimateD-iff [22].”
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (2024, unknown): “The denoiser network in video diffusion models can be instantiated by different neural architectures, such as U-Net [10, 24, 67, 111] or Transformers [5, 25, 80, 96].”
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, unknown): “With advancements in diffusion models and the availability of robust computational resources, a plethora of diffusion-based video generation models have emerged[4, 5, 8–10, 23, 25, 32].”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, methodology): “VideoCrafter[8] incorporates textual and visual features from CLIP as the input for cross-attention.”
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (2024, methodology): “Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods [19, 20, 7, 21, 22, 23, 24], and DiT based methods [6, 8, 5].”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “For text-to-video we compare VideoPoet with other recently published models, specifically: Show-1 [81], VideoCrafter [14] and Phenaki [65].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “Recent works [5, 15, 32, 144] shift their focus towards image-to-video generation for its finer content description and better scalability in training data.”
Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation (2023, unknown): “Magicvideo (Zhou et al., 2022), Video-Factory (Wang et al., 2023b), NUWA-XL (Yin et al., 2023), LaVie (Wang et al., 2023c), VideoCrafter (Chen et al., 2023) and Video LDM (Blattmann et al., 2023a) ground their models on latent-based VDMs.”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Our development is based on the open-source T2V model VideoCrafter [8] (@ 256 × 256 resolution) and T2I model Stable-Diffusion-v2.”