Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

By Jiangliu Wang, Jianbo Jiao, Linchao Bao, Shengfeng He, Yunhui Liu, Wei David Liu. 2019

223 citations recorded by the source corpus.

We address the problem of video representation learning without human-annotated labels. While previous efforts address the problem by designing novel self-supervised tasks using video data, the learned features are merely on a frame-by-frame basis, which are not applicable to many video analytic tasks where spatio-temporal features are prevailing. In this paper we propose a novel self-supervised approach to learn spatio-temporal features for video representation. Inspired by the success of two-stream approaches in video classification, we propose to learn visual features by regressing both motion and appearance statistics along spatial and temporal dimensions, given only the input video data. Specifically, we extract statistical concepts (fast-motion region and the corresponding dominant direction, spatio-temporal color diversity, dominant color, etc.) from simple patterns in both spatial and temporal domains. Unlike prior puzzles that are even hard for humans to solve, the proposed approach is consistent with human inherent visual habits and therefore easy to answer. We conduct extensive experiments with C3D to validate the effectiveness of our proposed approach. The experiments show that our approach can significantly improve the performance of C3D when applied to video classification tasks. Code is available at https://github.com/laura-wang/video_repres_mas.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, neutral): “Some of these tasks include: temporal ordering of video clips or frames [24, 46, 54, 86], predicting geometric transformations [38], maximizing the mutual information of multiple views [74], predicting motion and appearance [78], predicting the future, the past or a portion of masked input in the fe”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Examples include predicting the future frame [86], motion and appearance statistics [95], speed [9, 96] and encodings [62, 42, 43], sorting frames or video clips [58, 103, 49, 35].”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “While promising results have been achieved, some of the approaches leverage pre-computed motion channels [51, 15], e.”
Video Understanding as Machine Translation (2020, methodology): “Typical approaches use a pretext task defined by predicting transformations of the video data, such as temporal ordering [33, 57, 60, 22, 10], color or geometric transformations [51, 17], multiple views, motion/speed, and optical flow [48, 54, 16, 4].”
Learning Spatiotemporal Features via Video and Text Pair Discrimination (2020, methodology): “rder Veriﬁcation CaffeNet UCF101/HMDB51 50.2 18.1 OPN [31] Sequence order VGGNet UCF101/HMDB51 59.8 23.8 CMC [50] Optical ﬂow CaffeNet UCF101 55.3 - O3N [13] Odd-one-out AlexNet UCF101 60.3 32.5 MASN [55] Motion C3D Kinetics 61.2 33.4 COP [65] Clip order 3D ResNet10 UCF101 64.9 29.5 DPC [16] Predict”
RSPNet: Relative Speed Perception for Unsupervised Video Representation Learning (2020, methodology): “Speciﬁcally, with C3D, our method outperforms MAS [42] When we train longer ( i.e. , 1000 epochs), we can further improve the tkeop-1 accuracy to 93.7% and 64.7% for two datasets, respectively.”
Video Representation Learning with Visual Tempo Consistency (2020, background): “For instance, previous attempts learn video representations by predicting the correct order of shufﬂed frames [33], the arrow of time [46], and the frames and motion dynamics in the future [43, 15].”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “Other tasks make use of the temporal dimension of video to learn representations by predicting shufﬂed frames [38], the arrow of time [62], motion [60], clip order [65] or sorting sequences [32].”
Self-Supervised ECG Representation Learning for Emotion Recognition (2020, methodology): “This concept has been utilized in computer vision [21, 33–35], speech [36], natural language processing [37], and others.”
Self-Supervised Learning by Cross-Modal Audio-Video Clustering (2019, background): “In recent years, several approaches have extended image pre-text tasks to video [28, 68, 73].”