Everybody Dance Now

By Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros. 2018

857 citations recorded by the source corpus.

This paper presents a simple method for “do as I do” motion transfer: given a source video of a person dancing, we can transfer that performance to a novel (amateur) target after only a few minutes of the target subject performing standard moves. We approach this problem as video-to-video translation using pose as an intermediate representation. To transfer the motion, we extract poses from the source subject and apply the learned pose-to-appearance mapping to generate the target subject. We predict two consecutive frames for temporally coherent video results and introduce a separate pipeline for realistic face synthesis. Although our method is quite simple, it produces surprisingly compelling results (see video). This motivates us to also provide a forensics tool for reliable synthetic content detection, which is able to distinguish videos synthesized by our system from real data. In addition, we release a first-of-its-kind open-source dataset of videos that can be legally used for training and motion transfer.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Experimental and Cognitive Psychology, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Experimental and Cognitive Psychology

In Generative Models

Citations in the corpus

Rewriting a Deep Generative Model (2020, background): “At a practical level, deep generative models are increasingly useful for image and video synthesis [57,86,35,12].”
GAN Memory with No Forgetting (2020, background): “Generative adversarial networks (GANs) GANs have shown increasing power to synthesize highly realistic observations [32, 45, 51, 9, 33, 34], and have found wide applicability in various fields [39, 1, 19, 81, 83, 84, 12, 89, 37].”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, background): “GANs have made steady progress in unconditional image generation (Gulrajani et al., 2017; Karras et al., 2017, 2018), image-to-image translation (Isola et al., 2017; Zhu et al., 2017; Wang et al., 2018b) and video-to-video synthesis (Chan et al., 2018; Wang et al., 2018a).”
Video-to-Video Synthesis (2018, methodology): “Our method can change the clothing [79,81] for the same dancer (Figure 6 left) as well as transfer the visual appearance to new dancers (Figure 6 right) as explored in concurrent work [1,9,78].”
Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning (2019, background): “It is becoming easier to manipulate existing videos of people: replacing a person’s facial expressions or movements with those of another person [6, 7], and/or overlaying synthesized speech which imitates a person’s voice [8].”
SinGAN: Learning a Generative Model From a Single Natural Image (2019, background): “Single image deep models Several recent works proposed to “overfit” a deep model to a single training example [51, 60, 46, 7, 1].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Generative adversarial networks (GANs) [11] have shown a remarkable success across a broad range of applications in computer vision, graphics, and machine learning, e.g ., image generation [5, 20, 21], image-to-image translation [28, 33, 8], and video-to-video synthesis [43, 3, 6].”
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics (2019, background): “Although in recent years there have been many sophisticated algorithms for generating realistic synthetic face videos [9, 13, 46, 51, 26, 47, 37, 20, 23, 10, 21, 50], most of these have not been in mainstream as open-source software tools that anyone can use.”
Media Forensics and DeepFakes: An Overview (2020, background): “Beyond faces, some recent work addressed motion transfer: the target person dances following the movements transferred from a source dancer [46].”
First Order Motion Model for Image Animation (2020, background): “Traditional approaches for image animation and video re-targeting [6, 33, 13] were designed for specific domains such as faces [45, 42], human silhouettes [8, 37, 27] or gestures [31] and required a strong prior of the animated object.”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “Several methods can reduce the required number of input views by focusing on specific domains such as performance capture [8, 14, 21] and video re-animation [29, 34, 35, 62, 9].”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “[9] generate video sequences of dancing persons by first learning correspondences between frames and postures before adding appearance information.”
On Leveraging Pretrained GANs for Limited-Data Generation (2020, neutral): “…image (Hoffman et al., 2017; Ledig et al., 2017; Ak et al., 2019), text (Lin et al., 2017; Fedus et al., 2018; Wang & Wan, 2018; Wang et al., 2019), video (Mathieu et al., 2015; Wang et al., 2017; 2018a; Chan et al., 2019), and audio (Engel et al., 2019; Yamamoto et al., 2019; Kumar et al., 2019).”
Protecting World Leaders Against Deep Fakes (2019, background): “While convincing manipulations of digital images and videos have been demonstrated for several decades through the use of visual effects, recent advances in deep learning have led to a dramatic increase in the realism of fake content and the accessibility in which it can be created [28, 15, 30, 6, 2”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “Puppet-master deepfakes aim to hijack the source person’s expression, or even the full-body, [8] in a video, and to animate according to the impersonator’s desire.”
HoloGAN: Unsupervised Learning of 3D Representations From Natural Images (2019, background): “…adversarial networks (GANs), have achieved impressive results in generating images of high resolution and visual quality [1, 5, 27, 28, 62], while their conditional versions have achieved great progress in image-to-image translation [23, 50], image editing [11, 12, 60] or motion transfer [6, 30].”
Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “[9] addressed the problem of motion transfer, casting it within a per-frame image-to-image translation framework.”
Preserving integrity in online social networks (2020, background): “These advances have unleashed potential for harmful applications like face puppeteering [83, 136], speech manipulation [97], face transfer [115], full body manipulation [23], and other more general media manipulations.”
GAN Compression: Efficient Architectures for Interactive Conditional GANs (2020, background): “…image synthesis and enables many computer vision and graphics applications such as interactively creating an image from a user drawing [45], transferring the motion of a dancing video stream to a different person [57, 8, 1], or creating VR facial animation for remote social interaction [59].”
End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, background): “The recent years have witnessed the considerable progress of face forgery generation methods [2,4,11,18,20, 41, 42, 50, 53].”