Spatiotemporal Contrastive Video Representation Learning

By Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang, Serge J. Belongie, Yin Cui. 2020

583 citations recorded by the source corpus.

We present a self-supervised Contrastive Video Representation Learning (CVRL) method to learn spatiotemporal visual representations from unlabeled videos. Our representations are learned using a contrastive loss, where two augmented clips from the same short video are pulled together in the embedding space, while clips from different videos are pushed away. We study what makes for good data augmentations for video self-supervised learning and find that both spatial and temporal information are crucial. We carefully design data augmentations involving spatial and temporal cues. Concretely, we propose a temporally consistent spatial augmentation method to impose strong spatial augmentations on each frame of the video while maintaining the temporal consistency across frames. We also propose a sampling-based temporal augmentation method to avoid overly enforcing invariance on clips that are distant in time. On Kinetics-600, a linear classifier trained on the representations learned by CVRL achieves 70.4% top-1 accuracy with a 3D-ResNet-50 (R3D-50) backbone, outperforming ImageNet supervised pre-training by 15.7% and SimCLR unsupervised pre-training by 18.8% using the same inflated R3D-50. The performance of CVRL can be further improved to 72.9% with a larger R3D-152 (2× filters) backbone, significantly closing the gap between unsupervised and supervised video representation learning. Our code and models will be available at https://github.com/tensorflow/models/tree/master/official/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Video Representations from Textual Web Supervision (2020, methodology): “For example, prior approaches have successfully lever-aged supervision from clip and frame ordering [47, 16, 38, 66, 68, 33], geometry [17, 29], motion [53, 37], colorization [62], cycle consistency [14, 64], video prediction [44, 42, 61, 60, 63], tempo [69], and in-video consistency [22, 54].”
The Challenges of Continuous Self-Supervised Learning (2022, methodology): “coding [36, 54, 55], inpainting [59] and colorization [18, 34, 86], to methods that learn representations predictive of spatial context [19, 22, 53], temporal context [21, 47, 58, 61, 79, 80], or concurrent modalities like audio [4, 52, 57, 70], text [17, 23, 62] or speech [43, 44].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “[74] apply contrastive learning to videos with a temporal sampling strategy and temporally consistent spatial augmentation.”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, methodology): “We conclude by applying the training strategies to the Kinetics-400 video classiﬁcation task, using a 3D ResNet as the baseline architecture (Qian et al., 2020) (see Appendix G for experimental details).”
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, methodology): “Most selfsupervised visual contrastive learning methods [11, 12, 21, 28, 48] suffer from a large drop when removing strong spatial augmentations.”
Masked Feature Prediction for Self-Supervised Visual Pre-Training (2021, background): “The invariance is achieved by enforcing similarity over distorted views of one image while avoiding model collapse [8, 9, 15, 17, 34, 41, 44, 72, 93].”
A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (2021, methodology): “On K400 linear evaluation with the same data and R-50, Slow pathway [20] as backbone, our approach outperforms the previous best CVRL [72] by +5.”
CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding (2022, background): “In addition to that, with the success in exploitation of contrastive learning for image [7,13,18,33] and video [20, 40, 54] understanding, recent works have investigated self-supervised contrastive learning for point cloud understanding as well [21,44,59,68].”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “Specifically, most video methods extract the different views from a source video clip in a symmetric fashion with respect to time: all extracted views have the same temporal extent in the video [4, 23, 45, 67].”
Long-Short Temporal Contrastive Learning of Video Transformers (2021, background): “Just like in the image domain, more recent approaches [20,27,50] adopt contrastive learning objectives.”
Masked Autoencoders As Spatiotemporal Learners (2022, background): “Related topics include temporal coherence (‘slowness’) [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81], spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc .”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “Specifically, CMC extends positives to include different views, RGB and flow, of the same video clip, but does not introduce positives between clips; CVRL uses InfoNCE contrastive learning with video clips as the instances.”
Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity (2021, background): “2021), CVRL (Qian et al. 2021) relaxes the invariant constraint and develops a weighted temporal sampler to avoid excessive sampling of distant clips.”
A Survey on Contrastive Self-supervised Learning (2020, background): “[20], the framework contrasts the similarity between two positive samples to those of negative samples.”