RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning

By Peihao Chen, Deng Huang, Dongliang He, Xiang Long, Runhao Zeng, Shilei Wen, Mingkui Tan, Chuang Gan. 2020

143 citations recorded by the source corpus.

We study unsupervised video representation learning that seeks to learn both motion and appearance features from unlabeled video only, which can be reused for downstream tasks such as action recognition. This task, however, is extremely challenging due to 1) the highly complex spatial-temporal information in videos and 2) the lack of labeled data for training. Unlike representation learning for static images, it is difficult to construct a suitable self-supervised task to effectively model both motion and appearance features. More recently, several attempts have been made to learn video representation through video playback speed prediction. However, it is non-trivial to obtain precise speed labels for the videos. More critically, the learned models may tend to focus on motion patterns and thus may not learn appearance features well. In this paper, we observe that the relative playback speed is more consistent with motion patterns and thus provides more effective and stable supervision for representation learning. Therefore, we propose a new way to perceive the playback speed and exploit the relative speed between two video clips as labels. In this way, we are able to effectively perceive speed and learn better motion features. Moreover, to ensure the learning of appearance features, we further propose an appearance-focused task, where we enforce the model to perceive the appearance difference between two video clips. We show that jointly optimizing the two tasks consistently improves the performance on two downstream tasks (namely, action recognition and video retrieval) w.r.t the increasing pre-training epochs. Remarkably, for action recognition on the UCF101 dataset, we achieve 93.7% accuracy without the use of labeled data for pre-training, which outperforms the ImageNet supervised pre-trained model. Our code, pre-trained models, and supplementary materials can be found at https://github.com/PeihaoChen/RSPNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “In addition to a random prediction baseline, we compare against [21]’s RSPNet reported results (they use a 3DResNet-18 backbone pretrained on Kinetics400), and CLIP ViT-B/16 [89].”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, methodology): “, speed or playback rate (Benaim et al. 2020; Wang, Jiao, and Liu 2020; Chen et al. 2021; Yao et al. 2020), arrow of time (Wei et al.”
Efficient Test-Time Model Adaptation without Forgetting (2022, background): “Deep neural networks (DNNs) have achieved excellent performance in many challenging tasks, including image clas-siﬁcation (He et al., 2016; Tan & Le, 2019), video recognition (Wang et al., 2018; Chen et al., 2021), and many other areas (Choi et al., 2018; Fan et al., 2020; Xu et al., 2021).”