MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

By Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, Ying Shan. International Conference on Computer Graphics and Interactive Techniques, 2023

593 citations recorded by the source corpus.

Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods. Project page: https://wzhouxiff.github.io/projects/MotionCtrl/ .

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/44999d9bae3976d38e91b742c4541a21c7000260

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “For example, AnimateDiff [51] LoRA fine-tuned a video diffusion model with fixed types of camera motions, and MotionCtrl [52] conditioned the model on arbitrary specified camera trajectories.”
VACE: All-in-One Video Creation and Editing (2025, unknown): “Finally, we restructure the DiT model for VCU inputs, making it a versatile model for a wide range of video tasks.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, methodology): “MotionCtrl [13] is a camera-conditioned video diffusion model finetuned from SVD [17].”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “While the majority of existing methods produce videos without explicit controllability, two recent works [128, 139] introduce camera motion control to video generation.”
Gen3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (2025, unknown): “A key challenge in training G EN 3C is the lack of multi-view, dynamic, real-world video data, that provides the pairs of 3D cache and ground truth video for a novel camera trajectory.”
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (2023, background): “Diffusion models [23, 64] have made remarkable progress in text-conditioned image and video generation [4, 32, 52, 54, 58, 69, 71], attracting widespread attention in recent years.”