Self-Supervised Learning of Video-Induced Visual Invariances

By Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, Mario Lucic. 2019

67 citations recorded by the source corpus.

We propose a general framework for self-supervised learning of transferable visual representations based on Video-Induced Visual Invariances (VIVI). We consider the implicit hierarchy present in the videos and make use of (i) frame-level invariances (e.g. stability to color and contrast perturbations), (ii) shot/clip-level invariances (e.g. robustness to changes in object orientation and lighting conditions), and (iii) video-level invariances (semantic relationships of scenes across shots/clips), to define a holistic self-supervised loss. Training models using different variants of the proposed framework on videos from the YouTube-8M (YT8M) data set, we obtain state-of-the-art self-supervised transfer learning results on the 19 diverse downstream tasks of the Visual Task Adaptation Benchmark (VTAB), using only 1000 labels per task. We then show how to co-train our models jointly with labeled images, outperforming an ImageNet-pretrained ResNet-50 by 0.8 points with 10× fewer labeled images, as well as the previous best supervised model by 3.7 points using the full ImageNet data set.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Figure 2 decomposes the VTAB tasks into their respective groups, and compares to previous SOTA methods on this benchmark: BiT, VIVI – a ResNet co-trained on ImageNet and Youtube (Tschannen et al., 2020), and S4L – supervised plus semi-supervised learning on ImageNet (Zhai et al.”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “Specifically, three NoisyStudent [70] variants which use IMAGENET and unlabelled data from the JFT dataset, BiT (BigTransfer) [36] models that have been first trained on IMAGENET, IMAGENET-21K, or JFT and then transferred to IMAGENET by fine-tuning, and the Video-Induced Visual Invariance (VIVI) mod”
Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “76 0.09 86.4 [45] 87.4 [50] CIFAR-10 99.35 0.03 99.0 [17] - CIFAR-100 93.60 0.18 91.7 [43] - Pets 96.76 0.18 95.9 [17] 97.1 [33] Flowers 99.69 0.01 98.8 [43] 97.7 [33] VTAB (19 tasks) 76.65 0.11 71.7 [46] - We train both BiT-S and BiT-M for 90 epochs and decay the learning rate by a factor of 10 at ”
Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has started to spread over many other tasks in various other domains [73,86,60,76,47,37,74,81].”
What makes for good views for contrastive learning (2020, background): “Views are commonly different sensory signals, like photos and sounds [3], or different image channels [53] or slices in time [55], but may also be different “augmented" versions of the same data tensor [7].”
Toward Causal Representation Learning (2021, background): “Certain pretraining methods [36], [46], [55], [112], [196], [253] have shown that it is possible to achieve good results using only very few class labels by ﬁrst pretraining on a large unlabeled data set and then ﬁne-tuning on few labeled examples.”
Towards Causal Representation Learning (2021, background): “[196, 54, 112, 35, 45, 253] have shown that it is possible to achieve good results using only very few class labels by first pre-training on a large unlabeled dataset and then fine-tuning on few labeled examples.”
Large Scale Learning of General Visual Representations for Transfer (2019, background): “Overall, BiT-L with BiT-hyperparam outperforms the previously reported state-of-the-art on VTAB-1k [46].”
Contrastive learning of global and local features for medical image segmentation with limited annotations (2020, methodology): “Authors in [58] used domain-speciﬁc knowledge of videos while modeling the global contrastive loss.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “…CV. 1) SSL Models for Videos: SSL has garnered widespread usage across various applications, including video representation learning [203], [204], [205] and video retrieval [206]. a) Temporal information in videos: Various forms of temporal information in videos can be employed, encompassing…”
Big Self-Supervised Models Advance Medical Image Classification (2021, background): “Our Multi-Instance Contrastive Learning approach is related to previous works in video processing where multiple views naturally arising due to temporal variation [38, 42].”
Space-Time Correspondence as a Contrastive Random Walk (2020, background): “Strong image-level visual representations can be learned by heuristically choosing views that are close in space [100, 45, 4, 41, 16], in time [88, 82, 40, 97, 32] or both [46, 96], even when relying on noisy negative samples.”
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding (2020, result): “This links to recent work by Tschannen et al. [59], who show that temporal separation between frame embeddings inﬂuences the representation that is learned from videos.”
The Benchmark Lottery (2021, methodology): “The difference between models is on their architectures (e.g. WAE-GAN [Tolstikhin et al., 2017] vs. VIVI[Tschannen et al., 2020]), their sizes (e.g. ResNet-50 vs. ResNet-101 [Kolesnikov et al., 2019]), or the dataset they were pre-trained on (e.g. ResNet-50 pretrained on ImageNet-21k vs. ResNet-50…”