A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
By Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross B. Girshick, Kaiming He. 2021
302 citations recorded by the source corpus.
We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code will be made available at https://github.com/facebookresearch/SlowFast.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Hongsheng Li
- Trevor Darrell
- Christoph Feichtenhofer
- Zhe Gan
- Yoshua Bengio
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Yi Yang
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
In Representation & Self-Supervised Learning
- Yann LeCun
- Christoph Feichtenhofer
- Jean Ponce
- Mohit Bansal
- Kaiming He
- Hongsheng Li
- Xiatian Zhu
- Zeynep Akata
Citations in the corpus
- Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “Third, UL could supply strong pre-trained models that may perform on par with or even outperform supervised pre-training [12], [16], [154].”
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “, unsupervised learning [67, 20, 76], and video-text contrastive learning [64, 79, 78, 66].”
- Revisiting Feature Prediction for Learning Visual Representations from Video (2024, methodology): “Other works (Oord et al., 2018; Surís et al., 2021; Feichtenhofer et al., 2021) implement temporal invariance using noise-contrastive estimation (Gutmann and Hyvärinen, 2012).”
- Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “Concurrent to our work, [23] introduces ρBYOL which consists in directly applying BYOL, an image self-supervised technique, to video.”
- Masked Autoencoders As Spatiotemporal Learners (2022, background): “Related topics include temporal coherence (‘slowness’) [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81], spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc .”
- VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning (2021, methodology): “Previous constrastive learning methods [11, 13, 21] benefit from more negative samples.”