Latent Video Transformer

By Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, Evgeny Burnaev. 2020

140 citations recorded by the source corpus.

The video generation task can be formulated as a prediction of future video frames given some past frames. Recent generative models for videos face the problem of high computational requirements. Some models require up to 512 Tensor Processing Units for parallel training. In this work, we address this problem via modeling the dynamics in a latent space. After the transformation of frames into the latent space, our model predicts latent representation for the next frames in an autoregressive manner. We demonstrate the performance of our approach on BAIR Robot Pushing and Kinetics-600 datasets. The approach tends to reduce requirements to 8 Graphical Processing Units for training the models while maintaining comparable generation quality.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “In addition, action-conditioned samples are visually consistent with the input 3 SV2P (Babaeizadeh et al., 2017), SAVP (Lee et al., 2018), DVD-GAN-FP (Clark et al., 2019), Video Transformer (Weis-senborn et al., 2019), Latent Video Transformer (LVT) (Rakhimov et al., 2020), and TrIVD-GAN (Luc et al.”
NaturalSpeech: End-to-End Text-to-Speech Synthesis With Human-Level Quality (2022, methodology): “Inspired by image/video generation [21], [36], [37], [38], [39] that uses VQ-VAE [20], [40], [41] to compress high-dimensional image into low-dimensional representations to ease the generation [42], we leverage VAE [22] to compress high-dimensional speech x into frame-level representations z (i.e.,…”
CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers (2022, methodology): “Following the setup of Weissenborn et al. [33], Rakhimov et al. [17], we center-crop and downsample each frame to 64 × 64 to measure the FVD of the model.”
Taming Visually Guided Sound Generation (2021, background): “Applications of VQVAE for content generation include images [51, 61], audio [15, 37, 61, 71], and videos [49, 65].”
Phenaki: Variable Length Video Generation From Open Domain Textual Description (2022, methodology): “…networks [36, 42, 13, 50], variational based stochastic meth-ods [2, 10, 46, 3] and more recently by learning a discrete representation [49, 33, 31], auto-regressive models [51, 55, 28, 57], diffusion models [47, 14, 56, 19] ﬂow based models [24], and ﬁnally adversarial based methods [48,…”
GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions (2021, methodology): “Different from recent VQ-VAE based works such as Latent Video Transformer [20], our work focus on text-to-video generation task instead of video-to-video generation task.”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “Latent AViT [48] uses VQ-VAE [44] to com-RNN-RNN-RNN press the input images and apply AViT in the latent space to predict future frames.”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, methodology): “Rakhimov et al. [2020] (LVT) used transformers to predict the dynamics of video in latent space.”
Regeneration Learning: A Learning Paradigm for Data Generation (2023, background): “…or incurs too much computation cost, such as waveform generation in text-to-speech synthesis (Shen et al. 2018; Ren et al. 2019; Tan et al. 2021), and image/video/sound generation (Ramesh et al. 2021; Ding et al. 2021; Yan et al. 2021; Rakhimov et al. 2020; Rom-bach et al. 2022; Kreuk et al. 2022).”
Stochastic Image-to-Video Synthesis using cINNs (2021, methodology): “[62] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “In [33], hierarchical vector quantization is carried out in encoding and generating diverse image patches for inpainting; the work of [39] leverages quantized video frame representation to synthesize future frames.”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, methodology): “1, N ¨UWA signiﬁcantly outperforms 3 131 LVT [31] 1 126±3 SAVP [20] 2 116 DVD-GAN-FP [7] 1 110 Video Transformer (S) [44] 1 106±3 TriVD-GAN-FP [23] 1 103 CCVS [25] 1 99±2 Video Transformer (L) [44] 1 94±2 N ¨UWA 1 86.9 CogView [9] with FID-0 of 12.9 and CLIPSIM of 0.3429.”
Earthformer: Exploring Space-Time Transformers for Earth System Forecasting (2022, unknown): “Different from other papers that adopt Transformer for video prediction [19, 30], we generate the predictions in a non-auto-regressive fashion rather than an auto-regressive patch-by-patch fashion.”
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 (2021, unknown): “[52] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “For example, RNNs and LSTMs generate temporal noise vectors for an image generator [55,67,12,56,45,65]; transformers either directly generate pixel values [28,75] or indirectly predict latent tokens [46,16,52,77,36,81].”
Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, unknown): “In this case, we find our prosody encoder tends to index collapse [17], which means that some embedding vectors are close to a lot of encoders outputs and the model uses only a limited number of vectors from e.”