Self-Supervised Spatiotemporal Learning via Video Clip Order Prediction

By Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, Yueting Zhuang. 2019

481 citations recorded by the source corpus.

We propose a self-supervised spatiotemporal learning technique which leverages the chronological order of videos. Our method can learn the spatiotemporal representation of the video by predicting the order of shuffled clips from the video. The category of the video is not required, which gives our technique the potential to take advantage of infinite unannotated videos. There exist related works which use frames, while compared to frames, clips are more consistent with the video dynamics. Clips can help to reduce the uncertainty of orders and are more appropriate to learn a video representation. The 3D convolutional neural networks are utilized to extract features for clips, and these features are processed to predict the actual order. The learned representations are evaluated via nearest neighbor retrieval experiments. We also use the learned networks as the pre-trained models and finetune them on the action recognition task. Three types of 3D convolutional neural networks are tested in experiments, and we gain large improvements compared to existing self-supervised methods.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
Self-Supervised MultiModal Versatile Networks (2020, background): “In videos, works have also leveraged the temporal dimension [23, 46, 56, 91].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “ClipOrder [86] UCF R(2+1)D 30.”
Self-Supervised Learning of Pretext-Invariant Representations (2019, background): “For video data, these pretext tasks learn representations by ordering video frames [1, 18, 34, 41, 51, 79, 83], tracking [62, 77], or using cross-modal signals like audio [2, 3, 19, 36, 60, 61].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos (2020, methodology): “A variety of tasks, focused primarily on text-video modelling, have been applied to these datasets including: task segmentation [19–21], reference resolution [22, 23], action segmentation [5], video clip ordering [24–27], and action recognition [28–30].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Examples abound, including models for predicting the future [58, 43, 28], changing temporal sampling rates [73], sorting video frames or clips [42, 39, 72] and combining a few tasks [5].”
Long-Short Temporal Contrastive Learning of Video Transformers (2021, methodology): “Several methods for self-supervised video representation learning focused on predictive spatiotemporal ordering tasks [1, 23, 31, 33, 38, 43, 57, 58, 68, 70, 71].”
Multichannel Attention Refinement for Video Question Answering (2020, background): “3D CNNs [18, 41, 46] shows dramatic performance in action recognition recently when networks are trained on large datasets such as Kinetics [4].”
Speech2Action: Cross-Modal Supervision for Action Recognition (2020, unknown): “8 ClipOrder [48] R(2+1)D UCF101† [36] 30.”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, methodology): “VCOP (Xu et al. 2019) leverages the chronological order of videos and learn the spatiotemporal representations by predicting the order of shufﬂed clips from the video.”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Other tasks make use of the temporal dimension of video to learn representations by predicting shufﬂed frames [38], the arrow of time [62], motion [60], clip order [65] or sorting sequences [32].”
Oops! Predicting Unintentional Action in Video (2019, methodology): “Other approaches [62, 59, 75, 84] focus on leveraging large unlabeled datasets to learn visual representations useful for action anticipation.”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “When comparing the models that are only trained (both self-supervised and downstream finetune) on UCF101, e.g. OPN and VCOP, the proposed CoCLR (RGB network) obtains Top1 accuracy of 81.4 on UCF101 and 52.1 on HMDB, significantly outperforming all previous approaches.”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, unknown): “The pretext tasks include context prediction [20], playback rate perception [1], temporal clip orders prediction [46], etc .”
Revisiting Feature Prediction for Learning Visual Representations from Video (2024, unknown): “Finally, in Figure 8a we”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, methodology): “8 ClipOrder (Xu et al., 2019) UCF101 72.”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Related topics include temporal coherence (‘slowness’) [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81], spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc.”