Recovering Petaflops in Contrastive Semi-Supervised Learning of Visual Representations

By Mahmoud Assran, Nicolas Ballas, Lluís Castrejón, Michael G. Rabbat. 2020

3 citations recorded by the source corpus.

We investigate a strategy for improving the computational efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation, that aims to distinguish examples of different classes in addition to the self-supervised instance-wise pretext tasks. We find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches with significantly less computational effort. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that can significantly accelerate contrastive learning of visual representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Media Technology, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Media Technology

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, background): “2, 9 [3] Mahmoud Assran, Nicolas Ballas, Lluis Castrejon, and Michael Rabbat.”
Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (2021, methodology): “Following pre-training, we freeze the batch-norm layers, and ﬁne-tune the trunk of the network for 180 optimization steps on the available labeled samples using the supervised contrastive loss of [40], and do not apply any data-augmentations during this phase.”