Animating Arbitrary Objects via Deep Motion Transfer

By Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe. Computer Vision and Pattern Recognition, 2018

394 citations recorded by the source corpus.

This paper introduces a novel deep learning framework for image animation. Given an input image with a target object and a driving video sequence depicting a moving object, our framework generates a video in which the target object is animated according to the driving sequence. This is achieved through a deep architecture that decouples appearance and motion information. Our framework consists of three main modules: (i) a Keypoint Detector unsupervisely trained to extract object keypoints, (ii) a Dense Motion prediction network for generating dense heatmaps from sparse keypoints, in order to better encode motion information and (iii) a Motion Transfer Network, which uses the motion heatmaps and appearance information extracted from the input image to synthesize the output frames. We demonstrate the effectiveness of our method on several benchmark datasets, spanning a wide variety of object appearances, and show that our approach outperforms state-of-the-art image animation and video generation methods.

View Paper: https://www.semanticscholar.org/paper/4b473ad2bf0348c8a912c3af206fbf275f286320

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, unknown): “Conversely, explicit methods warp the source image to the target by either 2D optical flow [22, 26–28, 46, 52], 3D deformation field [3, 17, 36], or directly sawpping the face of target image [18].”
The Creation and Detection of Deepfakes (2020, background): “In contrast, the authors of [144] propose Monkey-Net: a self supervised network for driving an image with an arbitrary video sequence.”
Synthetic Data for Deep Learning (2019, background): “Similar ideas have been extended to animating still images [547], motion transfer [90], and image-to-image translation [384].”
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset (2021, methodology): “We reproduce previous landmark based works[33, 34, 49] on our HDTF dataset, and do quantitative and qualitative comparisons with them.”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “Video synthesis involves a wide range of tasks including video-to-video translation [79], image animation [69, 70], frame interpolation [56, 4], and video prediction.”
Motion Representations for Articulated Animation (2021, methodology): “For quantitative metrics, prior works [39, 28, 27] use video reconstruction accuracy as a proxy for image animation quality.”
Few-shot Video-to-Video Synthesis (2019, background): “• MonkeyNet [47] is proposed for transferring motions from a sequence to a still image.”
Learning Interactive Real-World Simulators (2023, unknown): “In generative modeling of videos, previous works have leveraged text prompts [64, 65], driving motions [66, 67], 3D geometries [68, 69], physical simulations [70], frequency information [71], and user annotations [72] to introduce movements into videos.”
First Order Motion Model for Image Animation (2020, methodology): “It’s important to note that one limitation of transferring relative motion is that we need to assume that the objects in S1 and D1 have similar poses (see [28]).”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “[14], [21] propose to encode motion information via keypoints learned in a self-supervised fashion and then warp the image according to the reference keypoint tra-”
DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors (2023, methodology): “To produce more realistic motion, reference-based methods [9, 37, 51, 55, 63– 65, 79 ] transfer motion or appearance information from Although existing approaches has achieved impressive performance, they primarily focus on animating motions in curated domains, particularly stochastic [5, 10, 14,…”
Thin-Plate Spline Motion Model for Image Animation (2022, methodology): “We also compared with other baseline methods FOMM [26], Monkey-Net [25] and X2Face [32] on video reconstruction.”
Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, unknown): “A straightforward idea is to generate the sign video from a sentence, which, however is a more challenging task involving various immature techniques, such as skeleton prediction [35], gesture generation [12] and temporal coherence ﬁdelity [38].”