Broaden Your Views for Self-Supervised Video Learning
By Adrià Recasens, Pauline Luc, Jean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Patraucean, Florent Altch'e, Michal Valko and 3 more. 2021
146 citations recorded by the source corpus.
Most successful self-supervised learning methods are trained to align the representations of two independent views from the data. State-of-the-art methods in video are inspired by image techniques, where these two views are similarly extracted by cropping and augmenting the resulting crop. However, these methods miss a crucial element in the video domain: time. We introduce BraVe, a self-supervised learning framework for video. In BraVe, one of the views has access to a narrow temporal window of the video while the other view has a broad access to the video content. Our models learn to generalise from the narrow view to the general content of the video. Furthermore, BraVe processes the views with different backbones, enabling the use of alternative augmentations or modalities into the broad view such as optical flow, randomly convolved RGB frames, audio or their combinations. We demonstrate that BraVe achieves state-of-the-art results in self-supervised representation learning on standard video and audio classification benchmarks including UCF101, HMDB51, Kinetics, ESC-50 and AudioSet.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Christoph Feichtenhofer
- Yanghao Li
- Haoqi Fan
- Kaiming He
- Boqing Gong
- Yin Cui
- Trevor Darrell
- Shih-Fu Chang
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Ming-Hsuan Yang
- Kaiming He
- Xiaogang Wang
- Andrew Zisserman
In Signal Processing
- Haoqi Fan
- Hassan Akbari
- Wei-Hong Chuang
- Christoph Feichtenhofer
- Yanghao Li
- Kaiming He
- Boqing Gong
- Yin Cui
Citations in the corpus
- VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “[80] learn to predict a broad view that spans a longer temporal context from a narrow view.”
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, unknown): “, video) to capture temporal information [30, 55, 78, 80].”
- Wav2CLIP: Learning Robust Audio Representations from Clip (2021, unknown): “We follow the original CLIP paper by using a contrastive loss [1], and experiment with adding multi-layer perceptron (MLP) as projection layers, similar to [6, 20, 21].”
- Masked Autoencoders As Spatiotemporal Learners (2022, background): “Related topics include temporal coherence (‘slowness’) [79, 25], future prediction [61, 72, 70, 45, 44, 71, 16], object motion [1, 75, 49, 76], temporal ordering [46, 23, 38, 78, 81], spatiotemporal contrast [58, 62, 30, 22, 51, 56], etc .”