Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs

By Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, Luc Van Gool. 2018

74 citations recorded by the source corpus.

The extension of image generation to video generation turns out to be a very difficult task, since the temporal dimension of videos introduces an extra challenge during the generation process. Besides, due to the limitation of memory and training stability, the generation becomes increasingly challenging with the increase of the resolution/duration of videos. In this work, we exploit the idea of progressive growing of Generative Adversarial Networks (GANs) for higher resolution video generation. In particular, we begin to produce video samples of low-resolution and short-duration, and then progressively increase both resolution and duration alone (or jointly) by adding new spatiotemporal convolutional layers to the current networks. Starting from the learning on a very raw-level spatial appearance and temporal movement of the video distribution, the proposed progressive method learns spatiotemporal information incrementally to generate higher resolution videos. Furthermore, we introduce a sliced version of Wasserstein GAN (SWGAN) loss to improve the distribution learning on the video data of high-dimension and mixed-spatiotemporal distribution. SWGAN loss replaces the distance between joint distributions by that of one-dimensional marginal distributions, making the loss easier to compute. We evaluate the proposed model on our collected face video dataset of 10,900 videos to generate photorealistic face videos of 256x256x32 resolution. In addition, our model also reaches a record inception score of 14.57 in unsupervised action recognition dataset UCF-101.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2018), Progressive VGAN (Acharya et al., 2018), TGAN-F (Kahembwe & Ramamoorthy, 2020), TGANv2 (Saito & Saito, 2018), DVD-GAN (Clark et al.”
Latent Video Transformer (2020, unknown): “…and unconditional video synthesis have been addressed for a long time and the solutions include recurrent models [15, 47, 48, 4], VAE-based models [8, 11, 25, 20, 9], autoregressive models [33, 39, 50, 22, 49, 19], normalizing ﬂows [24], GANs [27, 44, 35, 40, 1, 36, 7, 26] and optical ﬂow [32, 30].”
MAGVIT: Masked Generative Video Transformer (2022, unknown): “Early success in video synthesis has been made by GAN models [1,6,7,15,27,37,48,51, 56, 59, 66, 74].”
Structure and Content-Guided Video Synthesis with Diffusion Models (2023, background): “Unconditional video generation Generative adversarial networks (GANs) [12] can learn to synthesize videos based on specific training data [59, 45, 1, 56].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, background): “28, which is close to the upper bound shown in the “Real data” row [2,27].”
TGANv2: Efficient Training of Large Models for Video Generation with Multiple Subsampling Layers (2018, methodology): “ProgressiveGAN [23] generates high-resolution images by growing both the generator and the discriminator.”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, unknown): “Our generator tries (a lot) to reuse the content: we use classical content/motion decomposition with a single content code [1, 2], and (we believe) it gets exhausted in information for long videos.”
Adversarial Video Generation on Complex Datasets (2019, background): “The original objective was proposed by Goodfellow et al. (2014), and many improvements have since been suggested, mostly targeting improved training stability (Arjovsky et al., 2017; Zhang et al., 2018; Brock et al., 2019; Gulrajani et al., 2017; Miyato et al., 2018).”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “These methods achieve impressive performance gain by introducing novel neural operators like various RNNs [65–69,71] or transformers [48,70], delicate architectures like autoregressive [21,28,49,55,71] or normalizing flow [73], and applying distinct training strategies such as adversarial training [”