Self-supervised Video Representation Learning by Pace Prediction

By Jiangliu Wang, Jianbo Jiao, Yunhui Liu. 2020

263 citations recorded by the source corpus.

This paper addresses the problem of self-supervised video representation learning from a new perspective -- by video pace prediction. It stems from the observation that human visual system is sensitive to video pace, e.g., slow motion, a widely used technique in film making. Specifically, given a video played in natural pace, we randomly sample training clips in different paces and ask a neural network to identify the pace for each video clip. The assumption here is that the network can only succeed in such a pace reasoning task when it understands the underlying video content and learns representative spatio-temporal features. In addition, we further introduce contrastive learning to push the model towards discriminating different paces by maximizing the agreement on similar video content. To validate the effectiveness of the proposed method, we conduct extensive experiments on action recognition and video retrieval tasks with several alternative network architectures. Experimental evaluations show that our approach achieves state-of-the-art performance for self-supervised video representation learning across different network architectures and different benchmarks. The code and pre-trained models are available at this https URL.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, unknown): “Some works [9, 48, 43] try to predict the speciﬁc playback speed for each video.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, methodology): “Additionally, Wang et al. [214] addressed the challenge of self-supervised video representation learning through the lens of video pace prediction. b) Motions of objects in videos: Diba et al. [215] focused on SSL of motions in videos by employing dynamic motion ﬁlters to enhance motion…”
Long-Short Temporal Contrastive Learning of Video Transformers (2021, background): “Other approaches have leveraged temporal cues such as tempo and speed to define selfsupervised pretext tasks [5, 66].”
Video Representation Learning with Visual Tempo Consistency (2020, unknown): “7 PacePrediction [51] R(2+1)D-18 77.”
Unsupervised Single-Scene Semantic Segmentation for Earth Observation (2022, background): “Such approaches have been devised for different problems, e.g., image clustering [16], video analysis [17], and change detection in Earth observation images [18].”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, methodology): “Following the common evaluation protocol (Wang, Jiao, and Liu 2020; Wang et al. 2021b), we perform center cropping and average the scores on 10 uniformly sampled clips on validation splits of these datasets.”
Hard Negative Mixing for Contrastive Learning (2020, background): “…the sequential nature of the temporal dimension, e.g . future frame prediction and reconstruction [60], shufﬂing and then predicting or verifying the order of frames or clips [47, 38, 79], predicting the ”arrow of time” [74], pace [72] or predicting the “odd” element [21] from a set of clips.”