Predicting Video with VQVAE

By Jacob Walker, Ali Razavi, Aäron van den Oord. 2021

81 citations recorded by the source corpus.

In recent years, the task of video prediction-forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasized highly constrained datasets, we focus on very diverse, large-scale datasets such as Kinetics-600. We predict video at a higher resolution on unconstrained videos, 256x256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “While VQ-VAE-2 (Razavi et al., 2019) proposes using multi-scale hierarchical latents and SNAIL blocks (Chen et al., 2017) (and this setup has been applied to videos in (Walker et al., 2021)), the pipeline is inherently complicated and hard to reproduce.”
Video Diffusion Models (2022, neutral): “Video Transformer [59] 170 ± 5 DVD-GAN-FP [14] 69.1 ± 0.78 Video VQ-VAE [57] 64.3 ± 2.04 CCVS [31] 55 ± 1 TrIVD-GAN-FP [33] 25.74 ± 0.66 12.54 Transframer [35] 25”
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, methodology): “Some other recent works [20, 28, 33] apply VQ-VAE for the task of video prediction—forecasting future video frames given the past.”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “Method FVD ↓ Video Transformer [51] 170.0 ± 5.00 CogVideo [18] 109.2 DVD-GAN-FP [9] 69.1 ± 0.78 Video VQ-VAE [49] 64.3 ± 2.04 CCVS [28] 55.0 ± 1.00 TrIVD-GAN-FP [27] 25.7 ± 0.66 Transframer [31] 25.4 RaMViD [19] 16.5 Video Diffusion [17] 16.2 ± 0.34 Phenaki (Ours) 36.4 ± 0.19 Table 5.”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “Prior works mainly exploit deep generative models, including GANs [46, 30, 35, 36, 27, 26, 40], autoregressive models [44, 6], VAEs [41, 7], and normalizing ﬂows [17].”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, unknown): “2 [72] Jacob Walker, Ali Razavi, and Aäron van den Oord.”
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, methodology): “3.4) processes clips ci, c1i to build respective vector representations fi, f 1 i and an 3We do not use the Video-VQVAE [64] method since the image-trained VQVAE [51] has been pretrained on a very large image corpus and as a consequence cover a much more diverse set of visual scenes and elements.”
Genie: Generative Interactive Environments (2024, unknown): “…video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021; Lotter et al., 2017; Luc et al., 2020; Singer et al., 2023; Walker et al., 2021; Yan et al., 2021; Yu et al., 2023).”