Self-labelling via simultaneous clustering and representation learning

By Yuki M. Asano, Christian Rupprecht, Andrea Vedaldi. 2019

883 citations recorded by the source corpus.

Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard crossentropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet and yields the first self-supervised AlexNet that outperforms the supervised Pascal VOC detection baseline. Code and models are available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Another strand of methods use clustering to organize image examples (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; Li et al., 2021).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several variants allow automatic grouping of instances in the form of clustering [2, 7, 8, 21, 29, 35, 61, 65, 69].”
Self-supervised Pretraining of Visual Features in the Wild (2021, methodology): “Selfsupervised learning has a long history in computer vision with methods based on autoencoders [42, 51], clustering [1, 5, 11] or instance-level discrimination [3, 15, 21, 52].”
Exploring Simple Siamese Representation Learning (2020, methodology): “Like contrastive learning, clustering-based methods require either a memory bank [5, 6, 1], large batches [7], or a queue [7] to provide enough samples for clustering.”
ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “Image clusteringbased methods, such as DeepCluster [4], SwAV [5], and SeLa [2], generate labels by an unsupervised learning method initially and then use the subsequent assignments in a supervised way to update the weights of the network.”
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision (2021, background): “, 2018; 2019) or simultaneous clustering (Asano et al., 2019; Caron et al., 2020) methods studied in visual unsupervised learning research could be applied.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, positive): “DeepCluster-v2 and SeLa-v2, which are obtained by applying various training improvements introduced in other self-supervised learning papers to DeepCluster [6] and SeLa [2].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, neutral): “24M) 200 60.6 79.2 48.9 PCL (ours) R50 (24M) 200 61.5 82.3 49.2 SimCLR [9] R50-MLP (28M) 200 61.9 – – PCL v2 (ours) R50-MLP (28M) 200 67.6 85.4 50.3 LocalAgg [11] R50 (24M) 200 60.2 y– 50.1 SelfLabel [43] R50 (24M) 400 61.5 – – CPC [6] R101 (28M) – 48.7 – – CPCv2 [34] R170 w(303M) ˘200 65.9 – – CMC ”
Novel Class Discovery without Forgetting (2022, methodology): “This setting can be reduced to solving the following optimal transport problem [5,8]:”
Barlow Twins: Self-Supervised Learning via Redundancy Reduction (2021, methodology): “…compute ‘targets’ for the loss, and another distorted version of the sample to predict these targets, followed by an alternate optimization scheme like k-means in D EEP C LUS - TER (Caron et al., 2018) or non-differentiable operators in S W AV (Caron et al., 2020) and S E L A (Asano et al., 2020).”
Support-set bottlenecks for video-text representation learning (2020, background): “In the image and video domain, pretext tasks include colorization (Zhang et al., 2016), rotation (Gidaris et al., 2018), or clustering (Asano et al., 2020a;b; Caron et al., 2018; Ji et al., 2018), while in the natural language domain, masked language modeling (Devlin et al., 2019), and next word…”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Instead of viewing each sample as its own class, clustering-based methods [3, 46, 44, 27, 51, 6, 1, 45, 18, 5, 7] regroup them into clusters based on some similarity measure.”
How Well Do Self-Supervised Models Transfer? (2020, background): “On the other side, clustering-based approaches [4, 1] compare groups of images with similar features, sidestepping the intractability of instance discrimination.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “Most current self-supervised computer vision approaches can be categorised in contrastive learning [54, 25, 9, 23, 52, 56, 18], clustering methods [3, 67, 29, 5, 1, 20, 6, 7], asymmetric networks [22, 10] and feature-decorrelation methods [19, 63, 2, 27].”
Efficient Visual Pretraining with Contrastive Detection (2021, background): “While the benefits of instance-discrimination approaches have mostly been limited to pretraining from simple datasets such as ImageNet, clustering-based pretraining has proven very successful in leveraging large amounts of uncurated images for transfer learning [3, 6, 7, 20, 31].”
UP-DETR: Unsupervised Pre-training for Object Detection with Transformers (2020, background): “contrastive learning) attracts much attention in recent studies [16, 8, 14, 4, 6, 1], existing pretext tasks can not directly apply to pre-train the transformers of DETR.”
Multi-modal Self-Supervision from Generalized Data Transformations (2020, methodology): “We expect our method to yield even better results when combined with stronger architectures, as combining tasks has been shown to improve feature learning performance in multiple other papers [10, 67, 13].”
Residual Tuning: Toward Novel Category Discovery Without Labels (2022, neutral): “The core idea for tackling NCD is learning optimal clustering objectives [31]–[36], so as to make the unlabeled images discriminative in feature space.”