Thin-Plate Spline Motion Model for Image Animation
By Jian Zhao. Computer Vision and Pattern Recognition, 2022
304 citations recorded by the source corpus.
Image animation brings life to the static object in the source image according to the driving video. Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge. However, it remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images. In this paper, a new end-to-end unsupervised motion transfer framework is proposed to overcome such issues. Firstly, we propose thin-plate spline motion estimation to produce a more flexible optical flow, which warps the feature maps of the source image to the feature domain of the driving image. Secondly, in order to restore the missing regions more realistically, we leverage multi-resolution occlusion masks to achieve more effective feature fusion. Finally, additional auxiliary loss functions are designed to ensure that there is a clear division of labor in the network modules, encouraging the network to generate high-quality images. Our method11Our source code is publicly available: https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model. can animate a variety of objects, including talking faces, human bodies, and pixel animations. Experiments demonstrate that our method performs better on most benchmarks than the state of the art with visible improvements in motion-related metrics.
Subfields: Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/bc6bfac73406c4ad4d263a1e7c1a9923696abc3c
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, unknown): “Conversely, explicit methods warp the source image to the target by either 2D optical flow [22, 26–28, 46, 52], 3D deformation field [3, 17, 36], or directly sawpping the face of target image [18].”
- Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “The task of animating human images, a significant endeavor within the domain of video generation, aims to seamlessly create videos from one or multiple static images [6,30,35,36, 50,51,54,56].”
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Image animation[7, 13, 25, 33, 37–39, 57, 61, 64], aims to generate images or videos based on one or more input images.”
- SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation (2022, background): “Previous works also learn a shared intermediate motion representation from the source image and the target, which can be roughly divided into the landmark [39] and the unsupervised landmark-based meth-ods [14,34,40,48], 3DMM based methods [7,31,45] and the latent animation [23,42].”