Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos

By Yue Ma, Yin-Yin He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, Qifeng Chen. AAAI Conference on Artificial Intelligence, 2023

353 citations recorded by the source corpus.

Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e., image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models are available on https://follow-your-pose.github.io/.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/ee73edebd42626d9c2d91e35fd2ed3cdb0fb26d0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Yu Qiao
3. Chen Change Loy
4. Dacheng Tao
5. Ming-Hsuan Yang
6. Qi Tian
7. Yi Yang
8. Xiaogang Wang

In Control and Systems Engineering

1. Yu Qiao
2. Limin Wang
3. Ying Shan
4. Qi Tian
5. Wangmeng Zuo
6. Ziwei Liu
7. Luc Van Gool
8. Cordelia Schmid

Citations in the corpus

• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “Similar to adding controls in T2I [48, 61, 88, 93], introducing control signals in T2V, such as structure [17, 83], pose [44, 94], and Canny edge [39], has been increasingly receiving much attention.”
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “…have been proposed, leveraging control signals such as optical flow (Yin et al., 2023; Chen et al., 2023b; Shi et al., 2024), pose skeleton (Ma et al., 2023; Ruiz et al., 2023), and other multi-modal signals (Wang et al., 2024; Ruan et al., 2023), enabling more accurate control for guiding…”
• VBench: Comprehensive Benchmark Suite for Video Generative Models (2023, background): “Other guidance modalities are also available, including image-to-video [14, 16, 23, 122], video-to-video [12, 67, 80, 82, 121], and a variety of con-trol maps [15, 51, 58, 73, 103, 125, 126] such as pose, depth, and sketch.”
• HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, neutral): “This type of methods generates videos with other auxiliary controls, such as depth maps [27, 31], pose maps [89, 37, 83, 56], RGB images [5, 15, 61], or other guided motion videos [100, 86].”
• FateZero: Fusing Attentions for Zero-shot Text-based Video Editing (2023, background): “Diffusion-based models [19] can generate diverse and high-quality images [40,42,44] and videos [15,18,32,45,56] through text prompts.”
• Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Many studies[10, 16, 17, 23, 26, 30, 40, 52, 54] explore the augmentation of inter-frame attention modeling on the foundation of text-to-image (T2I) models to achieve video generation.”
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, neutral): “Recent works have embraced diffusion models for human-centric video generation [16] and animation [35].”
• ControlVideo: Training-free Controllable Text-to-Video Generation (2023, methodology): “Tune-A-Video [40] only preserves original human positions, while Text2Video-Zero [15] and Follow-Your-Pose [18] produce frames with appearance incoherence, e.g ., changing faces of iron man.”
• VACE: All-in-One Video Creation and Editing (2025, neutral): “…3) For controllable task, we use Control-A-Video [10], VideoComposer [68], and ControlVideo [79] under depth conditions, and compare Text2Video-Zero [31], ControlVideo [79], and Follow-Your-Pose [40] under pose conditions, as well as FLAT-17196 while being comparable to the metrics of Vidu 2.0.”