Video Pixel Networks

By Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, Koray Kavukcuoglu. 2016

446 citations recorded by the source corpus.

We propose a probabilistic video model, the Video Pixel Network (VPN), that estimates the discrete joint distribution of the raw pixel values in a video. The model and the neural architecture reflect the time, space and color structure of video tensors and encode it as a four-dimensional dependency chain. The VPN approaches the best possible performance on the Moving MNIST benchmark, a leap over the previous state of the art, and the generated videos show only minor deviations from the ground truth. The VPN also produces detailed samples on the action-conditional Robotic Pushing benchmark and generalizes to the motion of novel objects.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Neural Machine Translation in Linear Time (2016, methodology): “…have been proposed for modelling translation, such as encoder-decoder networks (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Kaiser & Bengio, 2016), networks with attentional pooling (Bahdanau et al., 2014) and two-dimensional networks (Kalchbrenner et al., 2016a).”
Denoising Diffusion Probabilistic Models (2020, background): “…(GANs), autoregressive models, ﬂows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [12, 25, 3, 55, 35, 23, 10, 30, 41, 54, 24, 31, 42], and there have been remarkable advances in energy-based modeling and score matching that have produced images…”
Axial Attention in Multidimensional Transformers (2019, background): “Neural network models in this family have achieved state-of-the-art log likelihoods on highdimensional image and video datasets (van den Oord et al., 2016a; Chen et al., 2018; Menick & Kalchbrenner, 2018; Parmar et al., 2018; Child et al., 2019; Weissenborn et al., 2019; Salimans et al., 2017; Kalch”
Scaling Autoregressive Video Models (2019, methodology): “We report results to compare with prior work on autoregressive video generation by Kalchbrenner et al. (2016), who achieve 0.92 bits/dim (0.64 nats/dim) with 2 frames of priming on each of the test splits (one with objects seen during training and one with novel objects).”
Neural Discrete Representation Learning (2017, background): “Recent advances in generative modelling of images [38, 12, 13, 22, 10], audio [37, 26] and videos [20, 11] have yielded impressive samples and applications [24, 18].”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, background): “WaveNet is one of a family of autoregressive deep generative models that have been applied with great success to data as diverse as text [18], images [17, 26, 19, 28], video [13], handwriting [8] as well as human speech and music.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “…begun to successfully model high dimensional raw observations from complex real-world datasets, from natural images and videos, to audio signals and natural language (Karras et al., 2017; Kalchbrenner et al., 2016b; van den Oord et al., 2016a; Kalchbrenner et al., 2016a; Vaswani et al., 2017).”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, background): “…trained by conventional maximum likelihood estimation (MLE) have produced superior scores on held-out data across a wide range of domains such as text (Vaswani et al., 2017; Wu et al., 2016), audio (van den Oord et al., 2016a), images (Parmar et al., 2018) and videos (Kalchbrenner et al., 2016).”
Efficient Neural Audio Synthesis (2018, background): “Copyright 2018 by the author(s). ages (van den Oord et al., 2016b; Reed et al., 2017) and videos (Kalchbrenner et al., 2017) and speech and music (van den Oord et al., 2016a; Mehri et al., 2016; Simon & Oore, 2017; Engel et al., 2017).”
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications (2017, methodology): “Besides being used for modeling images, the PixelCNN model was recently extended to model audio (van den Oord et al., 2016a), video (Kalchbrenner et al., 2016b) and text (Kalchbrenner et al., 2016a).”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “Video prediction (Srivastava et al., 2015; Finn et al., 2016; Kalchbrenner et al., 2017; Sønderby et al., 2020) can be seen as learning a generative model of future frames conditioned on the past frames.”
Generating Videos with Scene Dynamics (2016, background): “Our work is also related to efforts to predict the future in video [33, 22, 44, 51, 42, 17, 8, 55] as well as concurrent work in future generation [6, 15, 20, 50, 56, 43].”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Kalchbrenner et al. (2016) introduce an autoregressive video prediction model using gated CNNs and LSTMs. Recent approaches introduce stochasticity to the model to capture multiple futures (Babaeizadeh et al., 2017; Denton and Fergus, 2018).”
Value Prediction Network (2017, background): “Building an accurate observationprediction model is often very challenging when the observation space is large [23, 5, 13, 4] (e.”
Parallel Multiscale Autoregressive Density Estimation (2017, methodology): “For all PixelCNNs in the model, we used the same architecture as in (van den Oord et al., 2016b).”
Variational Lossy Autoencoder (2016, neutral): “These family of convolutional autoregressive models was further explored, and extended, for audio in WaveNet (Oord et al., 2016), video in Video Pixel Networks (Kalchbrenner et al., 2016b) and language in ByteNet (Kalchbrenner et al., 2016a).”
Adversarial Examples for Generative Models (2017, background): “2 Deep generative models, such as [5], learn to generate a variety of outputs, including handwritten digits, faces [6], realistic scenes [7], videos [8], 3D objects [9], and audio [10].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “This problem is characterized by many models’ failure to perform coherent long-term predictions, while performing accurate short-term predictions, even in trivial but partially observed environments [11, 13, 14].”