Unsupervised Learning of Video Representations using LSTMs

By Nitish Srivastava, Elman Mansimov, Ruslan Salakhutdinov. 2015

2,744 citations recorded by the source corpus.

We use Long Short Term Memory (LSTM) networks to learn representations of video sequences. Our model uses an encoder LSTM to map an input sequence into a fixed length representation. This representation is decoded using single or multiple decoder LSTMs to perform different tasks, such as reconstructing the input sequence, or predicting the future sequence. We experiment with two kinds of input sequences - patches of image pixels and high-level representations ("percepts") of video frames extracted using a pretrained convolutional net. We explore different design choices such as whether the decoder LSTMs should condition on the generated output. We analyze the outputs of the model qualitatively to see how well the model can extrapolate the learned video representation into the future and into the past. We further evaluate the representations by finetuning them for a supervised learning problem - human action recognition on the UCF-101 and HMDB-51 datasets. We show that the representations help improve classification accuracy, especially when there are only few training examples. Even models pretrained on unrelated datasets (300 hours of YouTube videos) can help action recognition performance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “…recently proposed sequence-to-sequence models employ conditional language models (Mikolov & Zweig, 2012) as their key component to solve diverse tasks like machine translation (Sutskever et al., 2014; Cho et al., 2014; Kalch-brenner et al., 2014) or video generation (Srivastava et al., 2015a).”
Semi-supervised Sequence Learning (2015, methodology): “[31], which has been successfully used f or machine translation [20, 10], text parsing [32], image captioning [ 34], video analysis [30], speech recognition [4, 3] and conversational modeling [27, 33].”
Sequence-Level Knowledge Distillation (2016, background): “…summarization (Rush et al., 2015), dialogue (Vinyals and Le, 2015; Serban et al., 2016; Li et al., 2016), NER/POS-tagging (Gillick et al., 2016), image captioning (Vinyals et al., 2015b; Xu et al., 2015), video generation (Srivastava et al., 2015), and speech recognition (Chan et al., 2015a).”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, neutral): “Some of the earliest work making use of temporal consistency used future frame prediction [56] as a pretext task.”
Scaling Autoregressive Video Models (2019, methodology): “Earlier work on video generation mostly focused on deterministic approaches (Srivastava et al., 2015; Carl Vondrick, 2016; Xingjian et al., 2015; Liu et al., 2017; Jia et al., 2016), which fail to capture the high degree of stochasticity inherent in video.”
Video Pixel Networks (2016, methodology): “Current approaches range from mean squared error models based on deep neural networks (Srivastava et al., 2015a; Oh et al., 2015), to models that predict quantized image patches (Ranzato et al., 2014), incorporate motion priors (Patraucean et al., 2015; Finn et al., 2016) or use adversarial losses…”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, background): “Video prediction (Srivastava et al., 2015; Finn et al., 2016; Kalchbrenner et al., 2017; Sønderby et al., 2020) can be seen as learning a generative model of future frames conditioned on the past frames.”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “, 2018), future-frame synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al.”
Generating Videos with Scene Dynamics (2016, background): “Finally, this paper is related to a growing body of work that capitalizes on large amounts of unlabeled video for visual recognition tasks [18, 46, 37, 13, 24, 25, 3, 32, 26, 27, 19, 41, 42, 1].”
Deep multi-scale video prediction beyond mean square error (2015, methodology): “In (Ranzato et al., 2014), the transformation back and forth between pixel and clustered spaces involves the averaging of 64 predictions of overlapping tilings of the image, in order to avoid a blockiness effect in the result.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, background): “Early works like (Mathieu et al., 2015; Srivastava et al., 2015; Han et al., 2019) aim to synthetic video frames through the image patches.”
Pointer Networks (2015, background): “Without the constraints, the predictions are bound to become blurry over longer sequences as shown in sequence-to-sequence models for videos [12].”
Learning to See by Moving (2015, methodology): “[40] aligned instances of the same class (from PASCALVOC-2012) in a global coordinate system and provide a rotation matrix (R) for each instance in the class.”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, neutral): “[28] applied a sequence-to-sequence learning framework [31] to a video domain, and showed that long short-term memory (LSTM) [12] networks are capable of generating video of bouncing handwritten digits.”
Generative Image Modeling Using Spatial LSTMs (2015, methodology): “It is therefore an ideal building block and may be used to extend other models such as DRAW [11] or video models [31, 37].”
Generating Images from Captions with Attention (2015, background): “The model can be viewed as a part of the sequence-to-sequence framework (Sutskever et al., 2014; Cho et al., 2014; Srivastava et al., 2015).”
Colorization as a Proxy Task for Visual Understanding (2017, methodology): “One way is to try to predict future frames, which is an analogous task to language modeling and often uses similar techniques based on RNNs and LSTMs [37, 34].”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, background): “While we have seen a rapid progress in video analysis [17, 31, 34], most existing work in VQA have focused primarily on images.”
Sequence to Sequence -- Video to Text (2015, methodology): “Another recent project [33] uses LSTMs to predict the future frame sequence from an encoding of the previous frames.”