First Order Motion Model for Image Animation

By Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe. 2020

1,235 citations recorded by the source corpus.

Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Generative Models

Citations in the corpus

Taming Transformers for High-Resolution Image Synthesis (2020, background): “This inductive bias towards local interactions thus leads to efficient computations, but the wide range of specialized layers which are introduced into CNNs to handle different synthesis tasks [46, 70, 59, 74, 73] suggest that this bias is often too restrictive.”
StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, background): “Other notable works (Nie et al. 2020; Siarohin et al. 2020; Zakharov et al. 2019) produce high quality image animations and manipulations.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, methodology): “Experiments and comparisons demonstrate the superiority of the proposed method compared to previous manipulation methods (either in image space [13], [14] or latent space[15], [16]), regarding both quality and controllability.”
Motion Representations for Articulated Animation (2021, methodology): “Follow on work [27] further modelled the motion around each keypoint with local, affine transformations, and introduced a generation module that both composites warped source image regions and inpaints occluded regions, to render the final image.”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, methodology): “Method PP [6] iPER [42] Tai-Chi [64] Human3.”
Creating, Using, Misusing, and Detecting Deep Fakes (2022, background): “VC predates deep fakes by 65 years, although recent efforts have incorporated deep learning (Sisman et al. 2020).”
Stochastic Image-to-Video Synthesis using cINNs (2021, background): “[70] Aliaksandr Siarohin, Stéphane Lathuiliére, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, methodology): “Synthesis Approaches To guarantee the diversity of synthesized images, we use four facial manipulation methods, including two face swapping methods: FSGAN [42] and FaceShifter [31], and two facial reenactment methods: First-order-motion [50] and IcFace [56].”
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, background): “Image animation[7, 13, 25, 33, 37–39, 57, 61, 64], aims to generate images or videos based on one or more input images.”
SofGAN: A Portrait Image Generator with Dynamic Styling (2020, background): “…to conduct dynamic image generation tasks such as free-viewpoint relighting [Chen et al. 2020], reenactment [Cao et al. 2013; Geng et al. 2018; Siarohin et al. 2019; Tewari et al. 2020; Thies et al. 2016], novel views synthesis [Chen et al. 2018; Lombardi et al. 2019; Mildenhall et al. 2020;…”
The Creation and Detection of Deepfakes (2020, methodology): “For example, Vid2Vid [168, 169], MocoGAN [162], and others [144, 145].”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “We evaluate our method on UCF-101 [34], Sky Time-lapse [43], and Taichi [28].”
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos (2020, background): “[29] is a flow-based approach and do not have disentanglement of background whereas [30] improves upon that method with occlusion masks that are inferred from flow based dense motions.”
Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction (2020, background): “[24].”
Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, methodology): “GAN based methods [35, 40, 44, 45] commonly employ warping functions to spatially transform the reference image according to input motion for generating sequential video frames.”
PIE (2020, methodology): “49 (45) Siarohin et al. [2019] 11.”