Video-to-Video Synthesis

By Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, Bryan Catanzaro. 2018

1,093 citations recorded by the source corpus.

We study the problem of video-to-video synthesis, whose goal is to learn a mapping function from an input source video (e.g., a sequence of semantic segmentation masks) to an output photorealistic video that precisely depicts the content of the source video. While its image counterpart, the image-to-image synthesis problem, is a popular topic, the video-to-video synthesis problem is less explored in the literature. Without understanding temporal dynamics, directly applying existing image synthesis approaches to an input video often results in temporally incoherent videos of low visual quality. In this paper, we propose a novel video-to-video synthesis approach under the generative adversarial learning framework. Through carefully-designed generator and discriminator architectures, coupled with a spatio-temporal adversarial objective, we achieve high-resolution, photorealistic, temporally coherent video results on a diverse set of input formats including segmentation masks, sketches, and poses. Experiments on multiple benchmarks show the advantage of our method compared to strong baselines. In particular, our model is capable of synthesizing 2K resolution videos of street scenes up to 30 seconds long, which significantly advances the state-of-the-art of video synthesis. Finally, we apply our approach to future video prediction, outperforming several state-of-the-art competing systems.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (2022, background): “Generating images from segmentation maps or scenes can be viewed as a conditional image synthesis task [71, 38, 24, 61, 62, 42].”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “…moving across multiple objects thereby benchmarking the ability of video models to capture object-robot interaction, object permanance, robot arm motion, etc. Translating videos to videos is another paradigm to think about video prediction with a prominent example being vid2vid (Wang et al., 2018).”
Semantic Image Synthesis With Spatially-Adaptive Normalization (2019, background): “Recent work has achieved photorealistic semantic image synthesis results [40, 44] on the Cityscapes dataset.”
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data (2019, background): “cycle consistency in domain translation [27], temporal coherence of consecutive frames in fluid simulation [28] and video translation [29].”
GAN Memory with No Forgetting (2020, background): “Generative adversarial networks (GANs) GANs have shown increasing power to synthesize highly realistic observations [32, 45, 51, 9, 33, 34], and have found wide applicability in various fields [39, 1, 19, 81, 83, 84, 12, 89, 37].”
MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis (2019, methodology): “Multi-Scale Architecture Following Wang et al. (2018b), we adopt a multi-scale architecture with 3 discriminators ( D 1 , D 2 , D 3 ) that have identical network structure but operate on different audio scales.”
Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, background): “Generative adversarial networks (GANs) [15] are popular models for sample generation, which have been the dominating paradigm for image generation [16, 17], image-to-image translation [18] and video-to-video synthesis [19].”
Seeing What a GAN Cannot Generate (2019, background): “Generative Adversarial Networks [15] have enabled many computer vision and graphics applications such as generation [7, 21, 22], image and video manipulation [19, 20, 30, 35, 39, 41, 52], object recognition [6, 42], and textto-image translation [33, 45, 49].”
Image2StyleGAN++: How to Edit the Embedded Images? (2019, background): “object detection [23], texture synthesis [22, 37, 31], image-to-image translation [16, 42, 28, 25] and video generation [33, 32, 35, 34].”
Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs (2020, background): “Generative adversarial networks (GANs) [11] have shown a remarkable success across a broad range of applications in computer vision, graphics, and machine learning, e.g ., image generation [5, 20, 21], image-to-image translation [28, 33, 8], and video-to-video synthesis [43, 3, 6].”
Media Forensics and DeepFakes: An Overview (2020, background): “It is also possible to generate a completely synthetic image or video using a segmentation map as input [36].”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Other than free-viewpoint rendering, there is another related active research field that focus on human motion retargeting either in 2D [2, 7, 35, 43, 54, 67, 68] or 3D [19, 20, 25, 32, 49, 53, 69, 74].”
Unsupervised Learning of Object Structure and Dynamics from Videos (2019, background): “Vid2Vid [27] proposed a video-to-video translation network from segmentation masks, edge masks and human pose.”
Low-Rank Subspaces in GANs (2021, background): “Generative Adversarial Networks (GANs), which can produce high-ﬁdelity images visually indistinguishable from real ones [19, 21, 4], have made tremendous progress in many real-world applications [17, 43, 35, 11, 42, 30, 8, 38].”
Style Generator Inversion for Image Enhancement and Animation (2019, background): “Recently, it has been shown that with ﬁducial point supervision [2, 28], facial pose can be transferred between a video and a still photograph.”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing (2020, background): “[21], GANs have shown promising results in various areas [44], such as unconditional image synthesis [21, 23, 31, 32, 33, 45, 55], image translation [8, 12, 26, 28, 42, 43, 52, 61, 67, 77, 96, 97], text-to-image translation [56, 84, 89], image processing [17, 18, 27, 35, 36, 37, 38, 40, 66, 72, 83, ”
The Creation and Detection of Deepfakes (2020, neutral): “ of an RNN T Ú :5; T Ú :6; T Ú :Ä; V A ç )4 7 5 )4 7 6 )4 7 Ä A ë :5; A ë :6; A ë :Ä; U æ ) Í ) Í ) Í Sample 1 Sample T L: Ú : ç ñ & Ï U Ï & Â U Â Image Disc. Video Disc. [21] Vid2Vid: L adv;L FM;L H æ :Ü?Å;áå æ :Ü; :Ü?Å ;áå 5 T æ :Ü?Å ;áå T .' * Intermediate Synth. Occlusion Masking W”