Motion Representations for Articulated Animation

By Aliaksandr Siarohin, Oliver J. Woodford, Jian Ren, Menglei Chai, Sergey Tulyakov. Computer Vision and Pattern Recognition, 2021

379 citations recorded by the source corpus.

We propose novel motion representations for animating articulated objects consisting of distinct parts. In a completely unsupervised manner, our method identifies object parts, tracks them in a driving video, and infers their motions by considering their principal axes. In contrast to the previous keypoint-based works, our method extracts meaningful and consistent regions, describing locations, shape, and pose. The regions correspond to semantically relevant and distinct object parts, that are more easily detected in frames of the driving video. To force decoupling of foreground from background, we model non-object related global motion with an additional affine transformation. To facilitate animation and prevent the leakage of the shape of the driving object, we disentangle shape and pose of objects in the region space. Our model1 can animate a variety of objects, surpassing previous methods by a large margin on existing benchmarks. We present a challenging new benchmark with high-resolution videos and show that the improvement is particularly pronounced when articulated objects are considered, reaching 96.6% user preference vs. the state of the art.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/1f122f73182db2f71fac27e014a23ae9b268138d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, background): “Moreover, some video generation methods still leverage the frame-level quality while neglecting the temporal information [72,84,100].”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “To produce more realistic motion, reference-based methods [9, 37, 51, 55, 63– 65, 79 ] transfer motion or appearance information from Although existing approaches has achieved impressive performance, they primarily focus on animating motions in curated domains, particularly stochastic [5, 10, 14,…”
Thin-Plate Spline Motion Model for Image Animation (2022, methodology): “As a comparison, unsupervised methods do not need to introduce a priori knowledge of the animated object during training [25, 26, 28, 32].”
Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos (2023, methodology): “FRAA [29] extends FOMM by a novel motion representation and synthesis framework for articulated characters using pose-dependent motion embedding.”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “The task of animating human images, a significant endeavor within the domain of video generation, aims to seamlessly create videos from one or multiple static images [6,30,35,36, 50,51,54,56].”
MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, neutral): “We perform a comprehensive comparison of MagicAnimate with several state-of-the-art methods for human image animation: (1) MRAA [28] and TPS [52] are state-of-the-art GAN-based animation approaches.”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, methodology): “We evaluate our method on three specific human video synthesis benchmarks (UBC fashion video dataset[59], TikTok dataset[20] and Ted-talk dataset[39]), using only the corresponding training datasets for each benchmark in the experiments.”
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, neutral): “It has been widely explored [13, 31, 34, 35, 39, 40, 42, 44, 47] recently.”
BlobGAN: Spatially Disentangled Scene Representations (2022, background): “For conditional generation of humans and objects, [79, 48, 47, 78] explore related blob-like intermediate representations to help generation.”