Video Representation Learning by Dense Predictive Coding

By Tengda Han, Weidi Xie, Andrew Zisserman. 2019

397 citations recorded by the source corpus.

The objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition. We make three contributions: First, we introduce the Dense Predictive Coding (DPC) framework for self-supervised representation learning on videos. This learns a dense encoding of spatio-temporal blocks by recurrently predicting future representations; Second, we propose a curriculum training scheme to predict further into the future with progressively less temporal context. This encourages the model to only encode slowly varying spatial-temporal signals, therefore leading to semantic representations; Third, we evaluate the approach by first training the DPC model on the Kinetics-400 dataset with self-supervised learning, and then finetuning the representation on a downstream task, i.e. action recognition. With single stream (RGB only), DPC pretrained representations achieve state-of-the-art self-supervised performance on both UCF101(75.7% top1 acc) and HMDB51(35.7% top1 acc), outperforming all previous learning methods by a significant margin, and approaching the performance of a baseline pre-trained on ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Are we done with ImageNet? (2020, neutral): “ielded corresponding gains on other tasks and modalities [6, 7, 8]. Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15]. As recent results continue to report systematic gains on ImageNet, we ask whether ”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Nevertheless, discriminative tasks useful for learning representations for action recognition, such as predicting whether a sequence of frames is played forward or backward [64], verifying whether the frames are ordered or shuffled [40], or predicting features corresponding to future frames [21], ca”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “, future frames in videos [25, 59], or operate on multiple views [73].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “To enhance this capability, we not only randomly sample negative cases but also re-sample video clips from the same video (Han et al., 2019).”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
Support-set bottlenecks for video-text representation learning (2020, background): “Discriminative approaches have proven to be particularly successful for image (Chen et al., 2020c; He et al., 2020; Misra & van der Maaten, 2020; Wu et al., 2018) and video (Han et al., 2019; Morgado et al., 2020; Patrick et al., 2020) representation learning.”
Audio-Visual Instance Discrimination with Cross-Modal Agreement (2020, methodology): “To compare to prior work, we follow [29, 41, 80] and evaluate visual representations on the UCF-101 [76] and HMDB-51 [42] datasets, by full network fine-tuning.”
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning (2020, background): “…tunable temperature parameter is critical to get diverse paraphrases for the supervised examples, consistent with previous work (Edunov et al., 2018; Xie et al., 2019), since commonly used beam search results in very regular sentences that do not provide diversity to the existing data distribution.”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “When compared to the very recent DPC algorithm [9] in Table 3, VIE shows improvement of more than 7.3%, even though DPC uses more frames (40 v.s. 16) as inputs and adds an additional recurrent cell between the action readout layer and its 3DResNet.”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Examples abound, including models for predicting the future [36, 27, 18], changing temporal sampling rates [46], and sorting video frames or clips [26, 24, 45].”
Self-Supervised Learning of Audio-Visual Objects from Video (2020, background): “In recent years, researchers have proposed a variety of self-supervised learning methods for learning representations from images [10, 17, 32, 33, 42, 45, 57, 58], videos [29, 30] and multimodal data [5, 40, 43, 47, 48].”
Video Representation Learning with Visual Tempo Consistency (2020, background): “For instance, previous attempts learn video representations by predicting the correct order of shufﬂed frames [33], the arrow of time [46], and the frames and motion dynamics in the future [43, 15].”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “…collapse by using a supervised action forecasting loss (Girdhar and Grauman, 2021), or by using the representations of distant clips as negative samples in a contrastive loss (Han et al., 2019, 2020; Tan et al., 2023), often focusing on small convolutional encoders (Han et al., 2019, 2020).”
Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, methodology): “on set duration training. To keep a balance between temporal receptive ﬁeld and GPU memory consumption, 8 frames are sampled from each video clip and the sampling stride is 4. Following the procedure [17,16], we perform multi-scale cropping, random horizontal ﬂip and color jittering for data augment”
Anticipative Video Transformer (2021, methodology): “[87] as well as follow ups [35, 36] that show that anticipating future visual representations is an effective form of self-supervision, though typically for traditional action recognition tasks.”
Contrastive Representation Learning: A Framework and Review (2020, background): “Expanding CPC into learning representations from natural videos,Dense Predictive Coding (DPC) [40] contrasts between local patches of the feature maps extracted from the past context with the local patches”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “The noise-contrastive loss function, which has recently been shown to yield good performance for images and videos [52, 19, 63, 41, 8, 35, 17, 56, 33], circumvents the need to implicitly specify what information needs to be discarded via a designed task.”