Video (language) modeling: a baseline for generative models of natural videos

By Marc'Aurelio Ranzato, Arthur Szlam, Joan Bruna, Michaël Mathieu, Ronan Collobert, Sumit Chopra. 2014

482 citations recorded by the source corpus.

We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed from the language modeling literature, and adapted to the vision domain by quantizing the space of image patches into a large dictionary. We demonstrate the approach on both a filling and a generation task. For the first time, we show that, after training on natural videos, such a model can predict non-trivial motions over short video sequences.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Videos with Scene Dynamics (2016, background): “Baseline: Since to our knowledge there are no existing large-scale generative models of video ([33] requires an input frame), we develop a simple but reasonable baseline for this task.”
Deep multi-scale video prediction beyond mean square error (2015, methodology): “In (Ranzato et al., 2014), the transformation back and forth between pixel and clustered spaces involves the averaging of 64 predictions of overlapping tilings of the image, in order to avoid a blockiness effect in the result.”
Learning to See by Moving (2015, background): “Past work in unsupervised learning has been dominated by approaches that pose feature learning as the problem of discovering compact and rich representations of images that are also sufficient to reconstruct the images [6, 3, 22, 32, 27, 30].”
Generative Image Modeling Using Spatial LSTMs (2015, methodology): “The model of Srivastava et al. [34] optimizes a squared error, which corresponds to a Gaussian assumption, while Ranzato et al. [28] try to side-step having to model pixel intensities by quantizing image patches.”
Unsupervised Learning of Video Representations using LSTMs (2015, methodology): “Recently, Ranzato et al. (2014) proposed a generative model for videos.”
Learning Image Representations Tied to Ego-Motion (2015, methodology): “hat couples ego-motor signals and video. To our knowledge, ours is the ﬁrst attempt to ground feature learning in physical activity. The limited prior work on unsupervised feature learning with video [20,22,19,7] learns only passively from observed scene dynamics, uninformed by explicit motor sensor”
Colorization as a Proxy Task for Visual Understanding (2017, background): “In this section we describe how to train colorization, revisiting some of the design decisions that were made with the goal of producing aesthetic color images and instead consider their impact on learning representations.”
Describing Videos by Exploiting Temporal Structure (2015, background): “Ranzato et al. in [24] also models a video clip with an RNN, however, after vector-quantizing image patches of the video clip.”
Dynamic Filter Networks (2016, methodology): “Ranzato et al. [18] use an encoder-decoder framework in a way similar to language modeling.”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, background): “Video Prediction Many works on video prediction learn a deterministic model conditioned on the current frame to predict the future ones (Ranzato et al., 2014; Srivastava et al., 2015).”
Unsupervised Learning of Long-Term Motion Dynamics for Videos (2017, methodology): “Ranzato et al. [21] proposed a generative model that uses a recurrent neural network to predict the next frame or interpolate between frames.”
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting (2015, background): “Recent advances in deep learning, especially recurrent neural network (RNN) and long short-term memory (LSTM) models [12, 11, 7, 8, 23, 13, 18, 21, 26], provide some useful insights on how to tackle this problem.”
Unsupervised Learning for Physical Interaction through Video Prediction (2016, background): “proposed a baseline for video prediction inspired by language models [21].”
Unsupervised Learning of Disentangled Representations from Video (2017, methodology): “[22] adopt a discrete vector quantization approach inspired by text models.”
Learning to Linearize Under Uncertainty (2015, background): “Recently there has been a flurry of work on learning features from video using varying degrees of supervision [14][12][13].”
Stochastic Variational Video Prediction (2017, background): “A number of prior works have addressed video frame prediction while assuming deterministic environments (Ranzato et al., 2014; Srivastava et al., 2015; Vondrick et al., 2015; Xingjian et al., 2015; Boots et al., 2014; Lotter et al., 2017).”
An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders (2016, methodology): “Keywords: Generative Models, Variational Autoencoders, Scene Understanding, Action Forecasting”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, neutral): “See, for example, [18, 21] and references therein.”