Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction.

By Longlong Jing, Xiaodong Yang, Jingen Liu, Yingli Tian. 2018

171 citations recorded by the source corpus.

The success of deep neural networks generally requires a vast amount of training data to be labeled, which is expensive and unfeasible in scale, especially for video collections. To alleviate this problem, in this paper, we propose 3DRotNet: a fully self-supervised approach to learn spatiotemporal features from unlabeled videos. A set of rotations are applied to all videos, and a pretext task is defined as prediction of these rotations. When accomplishing this task, 3DRotNet is actually trained to understand the semantic concepts and motions in videos. In other words, it learns a spatiotemporal video representation, which can be transferred to improve video understanding tasks in small datasets. Our extensive experiments successfully demonstrate the effectiveness of the proposed framework on action recognition, leading to significant improvements over the state-of-the-art self-supervised methods. With the self-supervised pre-trained 3DRotNet from large datasets, the recognition accuracy is boosted up by 20.4% on UCF101 and 16.7% on HMDB51 respectively, compared to the models trained from scratch.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, background): “The main goal of the rotation-prediction pretext task is to enforce the ConvNet to recognize high-level objects such as the horse’s leg or bird’s eyes [14,25].”
Unsupervised Learning From Video With Deep Neural Embeddings (2019, methodology): “In fact, the transfer performance of the single-frame unsupervised model trained on Kinetics is actually better than that of any model supervised on Kinetics.”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, background): “(Wang et al. 2019), identifying the rotation angle of video clips (Jing et al. 2018) or solving spatiotemporal jigsaw puzzles (Ahsan, Madhok, and Essa 2019; Kim, Cho, and Kweon 2019), etc.”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “(2) discriminative classification/regression, such as video order prediction [36, 14, 34, 57, 28], rotation transformation prediction [25], motion and appearance statistics regression [51], etc.”
Self-Supervised ECG Representation Learning for Emotion Recognition (2020, background): “For example, a self-supervised network was proposed in [41], where high-level spatiotemporal features were learned from unlabelled videos.”