Recycle-GAN: Unsupervised Video Retargeting

By Aayush Bansal, Shugao Ma, Deva Ramanan, Yaser Sheikh. European Conference on Computer Vision, 2018

319 citations recorded by the source corpus.

We introduce a data-driven approach for unsupervised video retargeting that translates content from one domain to another while preserving the style native to a domain, i.e., if contents of John Oliver's speech were to be transferred to Stephen Colbert, then the generated content/speech should be in Stephen Colbert's style. Our approach combines both spatial and temporal information along with adversarial losses for content translation and style preservation. In this work, we first study the advantages of using spatiotemporal constraints over spatial constraints for effective retargeting. We then demonstrate the proposed approach for the problems where information in both space and time matters such as face-to-face translation, flower-to-flower, wind and cloud synthesis, sunrise and sunset.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8ad3c383a79e85159098112127300dfd08c21319

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Text-based editing of talking-head video (2019, methodology): “Besides approaches that require a paired training corpus, unpaired video-to-video translation techniques[Bansal et al. 2018] only require two training videos.”
Adversarial Video Generation on Complex Datasets (2019, background): “Another extreme is occupied by strongly-conditioned models, including generation conditioned on another video for content transfer (Bansal et al., 2018; Zhou et al., 2019), per-frame segmentation masks (Wang et al., 2018), or pose information (Walker et al., 2017; Villegas et al., 2017; Yang et…”
A Survey on Generative Adversarial Networks: Variants, Applications, and Training (2020, methodology): “plications GANs models have also been utilized in many other video applications such as Pose-GAN (PGAN) [172] and Dual-Motion GAN (DMGAN) [173] for future frame (video) prediction; Recycle-GAN (RGAN) [174] for video retargeting, and Video Question Answering via Multi-Modal CGAN (VQAMM-CGAN) [175] fo”
Learning Correspondence From the Cycle-Consistency of Time (2019, background): “Forward-backward consistency is a specific case of cycle-consistency, which has been widely applied as a learning objective for 3D shape matching [21], image alignment [87, 89, 88], depth estimation [86, 14, 84], and image-to-image translation [90, 4].”
PIE (2020, methodology): “There has been a lot of research on person-specific techniques [Bansal et al. 2018; Kim et al. 2019, 2018; Thies et al. 2019, 2016; Wiles et al. 2018] that require a large training corpus of the target person as input.”
ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos (2020, background): “Our framework also does not require video-retargeting [22], [23].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Generative adversarial networks (GANs) [11] have shown a remarkable success across a broad range of applications in computer vision, graphics, and machine learning, e.g ., image generation [5, 20, 21], image-to-image translation [28, 33, 8], and video-to-video synthesis [43, 3, 6].”
Image-to-Image Translation: Methods and Applications (2021, background): “Moreover, image/video retargeting [188] enables the transfer of sequential content from one domain to another while preserving the style of the target domain.”
Animating Arbitrary Objects via Deep Motion Transfer (2018, background): “ering a single object category [46] or learning to translate videos from one specific domain to another [4, 22].”
Progressive Transfer Learning and Adversarial Domain Adaptation for Cross-Domain Skin Disease Classification (2020, methodology): “The method collapses when translating different domains with large structure changes, such as the reconstruction of a video of Barack Obama from Donald Trump [31].”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “A few distinguished improvements are i) a reduction in the amount of training data due to the introduction of un-paired self-supervised methods [231], ii) quick learning, which allows identity stealing using a single image [90, 92], iii) enhancements in visual details [85, 100], iv) improved tempora”
Detecting Photoshopped Faces by Scripting Photoshop (2019, background): “has mostly focused on “DeepFakes” and other GAN-based methods that may one day be able to convincingly simulate a real person’s appearance, movements, and facial expressions [35, 10, 7, 17], for now, such methods are prone to degeneracies and exhibit visible artifacts [25].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “Subject-dependent models [5, 88] can only work on specific persons since the model is only trained on the target person.”
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers (2018, background): “RecycleGAN [4] extends a concept of CycleGAN [58], and solves a video retargeting problem with two video datasets for which correspondences are not given.”
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations (2019, background): “[2] is able to transfer video content from one domain to another, which can be applied to face-to-face scenarios.”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Among them, some works [7, 40, 50, 92] proposed improved versions of the adversarial loss to stabilize GAN training and convergence, whereas others introduced additional losses [10, 58, 75, 83] for specific task learning.”
Behavior-Driven Synthesis of Human Dynamics (2021, background): “Some approaches synthesize videos directly in the pixel space [53, 2].”
The Creation and Detection of Deepfakes (2020, methodology): “, [15, 83, 169]).”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “A growing number of computer vision and machine learning applications require 2D human pose estimation as an input for their systems [56], [57], [58], [59], [60], [61], [62].”
Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation (2020, background): “On the other hand Cycle-GAN [18] and Recycle-GAN [19] and other popular examples of how cyclic training can be utilized to learn non-trivial cross-domain mapping, yielding reliable image-to-image transformation across different domains.”