Few-shot Video-to-Video Synthesis

By Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, Bryan Catanzaro. Neural Information Processing Systems, 2019

402 citations recorded by the source corpus.

Video-to-video synthesis (vid2vid) aims at converting an input semantic video, such as videos of human poses or segmentation masks, to an output photorealistic video. While the state-of-the-art of vid2vid has advanced significantly, existing approaches share two major limitations. First, they are data-hungry. Numerous images of a target human subject or a scene are required for training. Second, a learned model has limited generalization capability. A pose-to-human vid2vid model can only synthesize poses of the single person in the training set. It does not generalize to other humans that are not in the training set. To address the limitations, we propose a few-shot vid2vid framework, which learns to synthesize videos of previously unseen subjects or scenes by leveraging few example images of the target at test time. Our model achieves this few-shot generalization capability via a novel network weight generation module utilizing an attention mechanism. We conduct extensive experimental validations with comparisons to strong baselines using several large-scale video datasets including human-dancing videos, talking-head videos, and street-scene videos. The experimental results verify the effectiveness of the proposed framework in addressing the two limitations of existing vid2vid approaches.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bea92a69b48287caa3a25ff3dfe727bed8888348

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “object detection [23], texture synthesis [22, 37, 31], image-to-image translation [16, 42, 28, 25] and video generation [33, 32, 35, 34].”
StyleGAN2 Distillation for Feed-forward Image Manipulation (2020, methodology): “dulate the layer activations. So its usage is limited to the translation from segmentation maps. There are numerous follow-up works based on pix2pixHD architecture, including those working with video [7,52,53]. Unpaired Image-to-image translation The idea of applying cycle consistency to train on un”
PIE (2020, methodology): “…are image-based and cannot be controlled by intuitive parametric controls [Averbuch-Elor et al. 2017; Geng et al. 2018; Siarohin et al. 2019; Wang et al. 2019a; Zakharov et al. 2019], only enable editing of a single semantic parameter dimension, e.g., scene illumination [Meka et al. 2019;…”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, methodology): “…e.g ., transferring knowledge of a well-trained source model [45, 32, 44], learning meta-knowledge for quick adaptation to a target domain [24, 47, 42], using an auxiliary task to facilitate training [7, 26, 48, 49], improving an inference procedure of suboptimal models [2, 39, 29, 38], using an…”
AgileGAN (2021, background): “For example, Liu et al. [2019] and Wang et al. [2019] introduced a few-shot learning technique to perform appearance translation without needing a large dataset of specific style translation pairs.”
Motion Representations for Articulated Animation (2021, background): “Apart from some exceptions [35], these works further constrain the problem to animating a single instance of an object, such as a single face [16, 2] or a single human body [6, 25, 36], requiring retraining [2, 6, 25] or fine-tuning [40] for each new instance.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, methodology): “Video prompts can also be used for video generation as demonstrated in [93, 94].”
Media Forensics and DeepFakes: An Overview (2020, background): “These include content generation [22], [23], image/video synthesis from semantic labels [24], [25] or sketches [26] or text [27], changes of style and attributes [22], [28], [29], domain translation [30], up to expression transfer [25], face swapping typical of deepfakes [31] and talking-head video ”
Behavior-Driven Synthesis of Human Dynamics (2021, methodology): “A similar sequence-to-sequence translation task is performed in [57, 56, 36].”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, methodology): “Many generative adversarial methods, such as vid2vid [61, 60], leverage this type of input to guide synthesis.”
Pix2Video: Video Editing using Image Diffusion (2023, methodology): “In terms of conditional generation, several methods have utilized guidance channels such as segmentation masks or keypoints [55, 56].”
LOHO: Latent Optimization of Hairstyles via Orthogonalization (2021, background): “Generative models, in particular GANs, have been very successful across various computer vision applications such as image-to-image translation [15, 32, 40], video generation [34, 33, 9], and data augmentation for discriminative tasks such as object detection [24].”
Mimicry: Towards the Reproducibility of GAN Research (2020, background): “In particular, GANs are known for their creative applications in areas such as image synthesis [36, 17, 32, 18] image-to-image translation [42, 43, 23, 24], video-to-video synthesis [39, 38, 7], and even 3D view synthesis [29, 30, 40].”
Latent Video Transformer (2020, neutral): “Future video prediction, unconditional video synthesis or video to video translation [46, 31, 45].”
What comprises a good talking-head video generation?: A Survey and Benchmark (2020, methodology): “Then we training the network with the metric learning step using ArcLoss, which leads LRS3-TED VoxCeleb2 Real Chen [5] Baseline Wang [17] Wiles [18] Zakharov [20] Real Chen [5] Baseline Wang [17] Wiles [18] Zakharov [20] Top-1 72.62% 1.99% 3.85% 2.23% 1.93% 1.77% 42.46% 2.40% 3.04% 1.64% 1.87% 1.99%”
Single-Shot Freestyle Dance Reenactment (2020, methodology): “fsV2V[41] generates the entire video using a target image, OpenPose and DensePose data.”
InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs (2020, background): “Due to the great potential of GAN [2] in producing photo-realistic images, it has been widely applied to image editing [7], [8], superresolution [9], [10], image inpainting [11], [12], video synthesis [13], [14], etc.”
The Creation and Detection of Deepfakes (2020, methodology): “Several facial reenactment papers from Section 4.1 discuss body reenactment, too, for example, Vid2Vid [167, 168], MocoGAN [161], and others [143, 144].”