VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

By Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma. 2019

143 citations recorded by the source corpus.

Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modelling of video.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Imagen Video: High Definition Video Generation with Diffusion Models (2022, background): “…et al., 2016; Kalchbrenner et al., 2017; Babaeizadeh et al., 2021), latent-variable models with autoregressive priors (Mathieu et al., 2016; Vondrick et al., 2016; Babaeizadeh et al., 2018; Kumar et al., 2020), and more recently non-autoregressive latent-variable approaches (Gupta et al., 2022).”
MagicVideo: Efficient Video Generation With Latent Diffusion Models (2022, methodology): “Various video generation methods have been proposed in the past, including GAN-based meth-ods [47, 6, 44] and auto-regressive one [30, 17, 2, 19, 23, 15].”
Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer (2022, methodology): “Flow-based models [35] and ODE-based models [48] have also been used for video prediction.”
NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (2021, unknown): “MoCoGAN [37] 4 503 SVG-FP [8] 2 315 CNDA [12] 2 297 SV2P [1] 2 263 SRVP [13] 2 181 VideoFlow [18] 3 131 LVT [31] 1 126±3 SAVP [20] 2 116 DVD-GAN-FP [7] 1 110 Video Transformer (S) [44] 1 106±3 TriVD-GAN-FP [23] 1 103 CCVS [25] 1 99±2 Video Transformer (L) [44] 1 94±2”
Predicting Video with VQVAE (2021, background): “In comparison, we are aware of only a relatively small number of approaches which propose variational autoencoders [2, 60, 8], autoregressive models [20, 57], or ﬂow based approaches [22].”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “More recent work attempts to model uncertainty in pixel-space, often by learning a distribution of futures that can be sampled [8, 18, 21, 27, 29, 48, 51, 52, 62].”
Latent Video Diffusion Models for High-Fidelity Long Video Generation (2022, methodology): “Existing works capitalize on different types of generative models including GANs [40, 26, 27, 36, 35, 30, 2], VAEs [7, 44], autoregressive models [6, 42, 21], and normalizing ﬂows [17].”
Stochastic Image-to-Video Synthesis using cINNs (2021, unknown): “In particular, INNs can be implemented as normalizing flows [60], a special class of likelihood-based generative models which have recently been applied to various tasks, such as image synthesis [38, 2, 57], domain transfer [62, 61, 23, 82], superresolution [47, 82], and video synthesis [40].”
Adversarial Video Generation on Complex Datasets (2019, background): “…Lee et al., 2018; Hsieh et al., 2018), auto-regressive models (Ranzato et al., 2014; Srivastava et al., 2015; Kalchbrenner et al., 2017; Weissenborn et al., 2019), normalizing ﬂows (Kumar et al., 2019), and GANs (Mathieu et al., 2015; Vondrick et al., 2016; Saito et al., 2017; Saito & Saito, 2018).”