Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

By Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin. 2020

5,087 citations recorded by the source corpus.

Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or views) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a swapped prediction mechanism where we predict the cluster assignment of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of views with different resolutions in place of two full-resolution views, without increasing the memory or compute requirements much. We validate our findings by achieving 75.3% top-1 accuracy on ImageNet with ResNet-50, as well as surpassing supervised pretraining on all the considered transfer tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Another strand of methods use clustering to organize image examples (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; Li et al., 2021).”
Emerging Properties in Self-Supervised Vision Transformers (2021, positive): “However, the good performance with k-NN only emerge when combining certain components such as momentum encoder [26] and multi-crop augmentation [9].”
An Empirical Study of Training Self-Supervised Vision Transformers (2021, methodology): “We adopt a symmetrized loss [17, 7, 12]: ctr(q1, k2)+ctr(q2, k1).”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “We train these models online on a dataset of 2 B random internet images using the SwAV self-supervised approach [7].”
Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “For this purpose, we build upon recent works [3, 62] that exploit two losses to address both image classification and instance recognition, leveraging the “free” annotations provided by multiple data augmentations of a same instance, in the spirit of self-supervised learning [6, 9, 10, 25].”
Exploring Simple Siamese Representation Learning (2020, methodology): “Like contrastive learning, clustering-based methods require either a memory bank [5, 6, 1], large batches [7], or a queue [7] to provide enough samples for clustering.”
Projected GANs Converge Faster (2021, methodology): “More speciﬁcally, SwAV-FID computes the Fréchet distance in the penultimate layer of a ResNet-50 trained with the contrastive SwAV objective [6].”
High Fidelity Visualization of What Your Self-Supervised Representation Knows About (2021, positive): “Whereas the representation of the projector doesn’t seem to take into account any small transformation in the original conditioning outside the scale for Dino.”
ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “Image clusteringbased methods, such as DeepCluster [4], SwAV [5], and SeLa [2], generate labels by an unsupervised learning method initially and then use the subsequent assignments in a supervised way to update the weights of the network.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “Recently, self-supervised (Hénaff et al., 2019; Chen et al., 2020b; He et al., 2020; Misra & Maaten, 2020; Li et al., 2021; Grill et al., 2020; Caron et al., 2020) and semi-supervised learning (Yalniz et al.”
Unified Contrastive Learning in Image-Text-Label Space (2022, background): “Contrastive learning has laid the foundation for the best performing SSL models [3, 6, 8, 21, 24, 55, 56].”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, background): “, 2018; 2019) or simultaneous clustering (Asano et al., 2019; Caron et al., 2020) methods studied in visual unsupervised learning research could be applied.”
Prototypical Contrastive Learning of Unsupervised Representations (2020, critical): “In DeepCluster, the cluster assignments are considered as pseudo-labels and a classiﬁcation objective is optimized, which results in two weaknesses: (1) the high-dimensional features from the penultimate layer of a ConvNet are not optimal for clustering and need to be PCA-reduced; (2) an additional…”
Towards Label-Efficient Incremental Learning: A Survey (2023, background): “, 2021] and SwAV [Caron et al., 2020] for self-supervised pre-training.”
Online Continual Learning with Contrastive Vision Transformer (2022, positive): “Inspired by recent breakthroughs in self-supervised learning [16,11,30,72], we find that the knowledge learned by supervised contrastive learning [31,41,28] reveals greater robustness and transferability.”
Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay (2022, background): “Besides, inspired by the remarkable progress in self-supervised visual representation learning [1,2,6,7,19], [12] propose to enable continual learning without labeled data.”