Transformation-based Adversarial Video Prediction on Large-Scale Data

By Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, Karen Simonyan. 2020

95 citations recorded by the source corpus.

Recent breakthroughs in adversarial generative modeling have led to models capable of producing video samples of high quality, even on large and complex datasets of real-world video. In this work, we focus on the task of video prediction, where given a sequence of frames extracted from a video, the goal is to generate a plausible future sequence. We first improve the state of the art by performing a systematic empirical study of discriminator decompositions and proposing an architecture that yields faster convergence and higher performance than previous approaches. We then analyze recurrent units in the generator, and propose a novel recurrent unit which transforms its past hidden state according to predicted motion-like features, and refines it to to handle dis-occlusions, scene changes and other complex behavior. We show that this recurrent unit consistently outperforms previous designs. Our final model leads to a leap in the state-of-the-art performance, obtaining a test set Frechet Video Distance of 25.7, down from 69.2, on the large-scale Kinetics-600 dataset.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “In addition, action-conditioned samples are visually consistent with the input 3 SV2P (Babaeizadeh et al., 2017), SAVP (Lee et al., 2018), DVD-GAN-FP (Clark et al., 2019), Video Transformer (Weis-senborn et al., 2019), Latent Video Transformer (LVT) (Rakhimov et al., 2020), and TrIVD-GAN (Luc et al.”
Video Diffusion Models (2022, background): “Method FVD ↓ DVD-GAN [14] 109.8 VideoGPT [62] 103.3 TrIVD-GAN-FP [33] 103.3 Transframer [35] 100 CCVS [31] 99 VideoTransformer [59] 94 FitVid [4] 93.6 NUWA [61] 86.9 Video Diffusion (ours) ancestral sampler, 512 steps 68.19 Langevin sampler, 256 steps 66.92 Table 3: Video prediction on Kinetics-600.”
Predicting Video with VQVAE (2021, background): “We ﬁnd performance exceeds [7] but not necessarily [27].”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, unknown): “1, N ¨UWA signiﬁcantly outperforms 3 131 LVT [31] 1 126±3 SAVP [20] 2 116 DVD-GAN-FP [7] 1 110 Video Transformer (S) [44] 1 106±3 TriVD-GAN-FP [23] 1 103 CCVS [25] 1 99±2 Video Transformer (L) [44] 1 94±2 N ¨UWA 1 86.9 CogView [9] with FID-0 of 12.9 and CLIPSIM of 0.3429.”
Transframer: Arbitrary Frame Prediction with Generative Models (2022, background): “4, improving over the previous state-of-the-art model TrIVD-GAN-FP [40].”
MAGVIT: Masked Generative Video Transformer (2022, methodology): “We evaluate on the large-scale SSv2 dataset, where MAGVIT needs to synthesize 174 basic actions with everyday objects.”
A Review on Deep Learning Techniques for Video Prediction (2020, unknown): “[164] proposed the Transformation-based & TrIple Video Discriminator GAN (TrIVD-GAN-FP) featuring a novel recurrent unit that computes the parameters”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “These methods achieve impressive performance gain by introducing novel neural operators like various RNNs [65–69,71] or transformers [48,70], delicate architectures like autoregressive [21,28,49,55,71] or normalizing flow [73], and applying distinct training strategies such as adversarial training [”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “1 Training DVD-GAN [12] or DVD-GAN-FP [40] on 16-frame videos requires 32-512 TPU replicas and 12-96 hours.”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “TriVD-GAN Luc et al. [2020] proposes a novel recurrent unit in the generator to handle more complex dynamics, while DIGAN Yu et al. [2022] uses implicit neural representations in the generator.”
Photorealistic Video Generation with Diffusion Models (2023, methodology): “Specifically, the Method K600 FVD ↓ UCF FVD ↓ params. steps TrIVD-GAN-FP [51] model is conditioned using c where m fp is a binary mask.”
Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “[47] Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan.”
Genie: Generative Interactive Environments (2024, unknown): “…video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021; Lotter et al., 2017; Luc et al., 2020; Singer et al., 2023; Walker et al., 2021; Yan et al., 2021; Yu et al., 2023).”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, background): “Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
Latent Video Transformer (2020, unknown): “Even for short videos ( 16 frames) of low resolution, neural networks require up to 512 Tensor Processing Units (TPUs) [26] for parallel training.”