Improved Conditional VRNNs for Video Prediction

By Lluís Castrejón, Nicolas Ballas, Aaron C. Courville. 2019

180 citations recorded by the source corpus.

Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational Auto-Encoder. While VAEs can handle uncertainty and model multiple possible future outcomes, they have a tendency to produce blurry predictions. In this work we argue that this is a sign of underfitting. To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models. Our approach relies on a hierarchy of latent variables, which defines a family of flexible prior and posterior distributions in order to better model the probability of future sequences. We validate our proposal through a series of ablation experiments and compare our approach to current state-of-the-art latent variable models. Our method performs favorably under several metrics in three different datasets.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models (2023, background): “20 [8] Lluis Castrejon, Nicolas Ballas, and Aaron Courville.”
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis (2021, methodology): “However, also more complex models based on transformers [75], pixel-level autoregression [51, 40, 45, 16, 70, 10], factorization of dynamics and content [53, 24] and image warping using optical flow [71, 44, 25] have been proposed.”
A Review on Deep Learning Techniques for Video Prediction (2020, methodology): “Many approaches were taken to leverage the advantages of variational inference: combined adversarial training with VAEs [55], and others incorporated latent probabilistic variables into deterministic models, such as Variational Recurrent Neural Networks (VRNNs) [97], [98] and Variational Encoder-Dec”
A review of convolutional neural networks in computer vision (2024, unknown): “The CNN encoder and RNN decoder are combined in a variable generation framework in conditional VRNN (Castrejon et al. 2019).”
Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020, methodology): “To improve predictions, recent methods use adversarial training [40, 62, 29], stochastic models [7, 41], constraint predictions by using geometric knowledge [16, 24, 75] or by disentangling factors of variation [60, 58, 12, 21].”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “These VAE techniques are used for tasks such as future frame prediction (Castrejon et al., 2019), image segmentation (Kohl et al., 2018), generating speech (Chung et al., 2015) and music (Dhariwal et al., 2020), as well as model-based reinforcement learning (Hafner et al., 2019a).”
Diffusion Probabilistic Modeling for Video Generation (2022, methodology): “Our method is also better at probabilistic forecasting than modern GAN and VAE baselines such as IVRNN, SVG-LP, RetroGAN, DVDGAN and FutureGAN (Castrejon et al., 2019; Denton & Fergus, 2018; Kwon & Park, 2019; Clark et al., 2019; Aigner & Körner, 2018).”
SimVP: Simpler yet Better Video Prediction (2022, methodology): “Conditional VRNN [6] combines CNN encoder and RNN decoder in a variational generating framework.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “…based on variational inference (Chung et al., 2015; Rezende et al., 2014; Kingma and Welling, 2014), including (Babaeizadeh et al., 2018; Denton and Fergus, 2018; Lee et al., 2018; Li and Mandt, 2018; He et al., 2018) and large-scale models such as (Castrejon et al., 2019; Villegas et al., 2019).”
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning (2021, unknown): “inference with 2D recurrence [35], [51], [52], [53], [54], [55], [56].”