I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models

By Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Z. Qin, Xiang Wang, Deli Zhao, Jingren Zhou. arXiv.org, 2023

375 citations recorded by the source corpus.

Video synthesis has recently made remarkable strides benefiting from the rapid development of diffusion models. However, it still encounters challenges in terms of semantic accuracy, clarity and spatio-temporal continuity. They primarily arise from the scarcity of well-aligned text-video data and the complex inherent structure of videos, making it difficult for the model to simultaneously ensure semantic and qualitative excellence. In this report, we propose a cascaded I2VGen-XL approach that enhances model performance by decoupling these two factors and ensures the alignment of the input data by utilizing static images as a form of crucial guidance. I2VGen-XL consists of two stages: i) the base stage guarantees coherent semantics and preserves content from input images by using two hierarchical encoders, and ii) the refinement stage enhances the video's details by incorporating an additional brief text and improves the resolution to 1280$\times$720. To improve the diversity, we collect around 35 million single-shot text-video pairs and 6 billion text-image pairs to optimize the model. By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos. Through extensive experiments, we have investigated the underlying principles of I2VGen-XL and compared it with current top methods, which can demonstrate its effectiveness on diverse data. The source code and models will be publicly available at \url{https://i2vgen-xl.github.io}.

Subfields: Biophysics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9b86ce1bde87b304141641b49299f4d0f1f7ba1d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Computer Vision and Pattern Recognition

Citations in the corpus

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, neutral): “Several studies extend text-to-video methodologies to image-to-video[8, 12, 48, 63].”
VACE: All-in-One Video Creation and Editing (2025, neutral): “With the rapid development of image [2, 7, 16, 18, 58, 59] and video [22, 32, 73, 77] generation models, they are being used to create high-quality visual content and are widely applied in fields such as advertising, film special effects, game development, and animation production [13, 43–45 , 55 ].”
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets (2023, methodology): “Similar to the image domain, diffusion-based approaches can be mainly separated into cascaded approaches [41] following [29, 39] and latent diffusion models [9, 113, 115] translating the approach of Rombach et al. [71] to the video domain.”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “Many recently video diffusion models are the T2V models (Karras et al., 2023; Ruan et al., 2023; Zhang et al., 2023b; He et al., 2022; Chen et al., 2023a; Hong et al., 2022; Yang et al., 2024b).”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “The trade-off between perceptual quality and motion intensity has been widely acknowledged in video generation [3, 32, 73, 144], and our case is no exception.”