VideoComposer: Compositional Video Synthesis with Motion Controllability

By Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, Jingren Zhou. Neural Information Processing Systems, 2023

542 citations recorded by the source corpus.

The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models will be publicly available at https://videocomposer.github.io.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f02ea7a18f00859d9ea1b321e3385ae7d0170639

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

1. Ying Shan
2. Qifeng Chen
3. Yu Qiao
4. Dayou Chen
5. Deli Zhao
6. Jingren Zhou
7. Ziwei Liu
8. Xiang Wang

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Chen Change Loy
4. Ming-Hsuan Yang
5. Dacheng Tao
6. Ziwei Liu
7. Ying Shan
8. Xiaoou Tang

Citations in the corpus

• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “VideoComposer[48] incorporates images into the diffusion input during training as a conditional control.”
• Lumiere: A Space-Time Diffusion Model for Video Generation (2024, unknown): “We empirically found that initializing these modules such that they perform nearest-neighbor down-and up-sampling operations results with a good starting point (in terms of the loss function).”
• I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models (2023, methodology): “For a video x ∈ F × H × W × 3 , we follow VideoComposer [46] to use the encoder of a pretrained VQGAN [9] to compress it into a low-dimensional latent representation z = E ( x ) , where z ∈ R F × h × w × c .”
• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “Other guidance modalities are also available, including image-to-video [14, 16, 23, 122], video-to-video [12, 67, 80, 82, 121], and a variety of con-trol maps [15, 51, 58, 73, 103, 125, 126] such as pose, depth, and sketch.”
• ModelScope Text-to-Video Technical Report (2023, background): “This encompasses textual content [40, 5, 56], visual content [39, 24, 48, 44, 27, 71, 16, 38, 51, 59] and auditory content [4, 32, 31].”
• VACE: All-in-One Video Creation and Editing (2025, unknown): “…[9, 22, 64] generation in the field, there has been a proliferation of down-stream tasks and applications, such as repainting [3, 82], editing [4, 42, 68, 70, 75], controllable generation [30, 76], frame reference generation [20, 73], and ID-referenced video synthesis [11, 35, 47, 74].”
• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “We evaluate our method against VideoComposer [77] and I2VGen-XL [12], with the quantitative results presented in Table 1.”
• VideoCrafter1: Open Diffusion Models for High-Quality Video Generation (2023, background): “VideoComposer [51] is a recently released model for compositional video generation that supports text-image-to-video synthesis.”
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, unknown): “Recaprture (Zhang et al., 2024) enables video-to-video camera control, effectively modifying viewpoints in existing content.”
• MotionCtrl: A Unified and Flexible Motion Controller for Video Generation (2023, methodology): “In contrast, VideoComposer [25] relies on dense motion vectors and mistakenly captures a door’s shape of the reference video, resulting in an unnatural Eiffel Tower.”