ClusterFit: Improving Generalization of Visual Representations

By Xueting Yan, Ishan Misra, Abhinav Kumar Gupta, Deepti Ghadiyaram, Dhruv Kumar Mahajan. 2019

140 citations recorded by the source corpus.

Pre-training convolutional neural networks with weakly-supervised and self-supervised strategies is becoming increasingly popular for several computer vision tasks. However, due to the lack of strong discriminative signals, these learned representations may overfit to the pre-training objective (e.g., hashtag prediction) and not generalize well to downstream tasks. In this work, we present a simple strategy - ClusterFit to improve the robustness of the visual representations learned during pre-training. Given a dataset, we (a) cluster its features extracted from a pre-trained network using k-means and (b) re-train a new network from scratch on this dataset using cluster assignments as pseudo-labels. We empirically show that clustering helps reduce the pre-training task-specific information from the extracted features thereby minimizing overfitting to the same. Our approach is extensible to different pre-training frameworks -- weak- and self-supervised, modalities -- images and videos, and pre-training tasks -- object and action classification. Through extensive transfer learning experiments on 11 different target datasets of varied vocabularies and granularities, we show that ClusterFit significantly improves the representation quality compared to the state-of-the-art large-scale (millions / billions) weakly-supervised image and video models and self-supervised image models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Grafit: Learning fine-grained image representations with coarse labels (2020, methodology): “Table 7 compares a ResNet-50 pretrained on ImageNet with Graﬁt, SNCA+, ClusterFit [67] and our baseline on ﬁve transfer learning benchmarks.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Our work is also related to clustering-based methods [2, 3, 6, 7, 17, 25, 50, 53, 54, 59].”
ImageNet-21K Pretraining for the Masses (2021, methodology): “Knowledge distillation (KD) is a known method to improve not only upstream, but also downstream 250 results [57, 58, 60].”
Self-Supervised Learning: Generative or Contrastive (2020, methodology): “Clus-terFit [146] introduces a cluster prediction ﬁne-tuning stage similar to DeepCluster between the above two stages, which improves the representation’s performance on downstream classiﬁcation evaluation.”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Instead of viewing each sample as its own class, clustering-based methods [3, 46, 44, 27, 51, 6, 1, 45, 18, 5, 7] regroup them into clusters based on some similarity measure.”
With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations (2021, background): “Methods that use prototypes/clusters [1, 2, 5, 6, 7, 23, 33, 53, 56, 57, 58, 60] also attempt to learn features by associating multiple samples with the same cluster.”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since computing all the pairwise comparisons on a large dataset is challenging, some clustering-based methods turn to discriminate between groups of images with similar features instead of individual images [2, 5, 12, 13, 64, 104, 119, 121, 122].”
A Survey of Deep Learning for Scientific Discovery (2020, background): “…tasks, some examples are: taking an image level label (for classiﬁcation) and automatically inferring pixel level labels for segmentation [171], clustering hidden representations computed by a pretrained network as pseudo-labels [255], or taking Instagram tags as labels [143] for pretraining.”
Exploring Visual Engagement Signals for Representation Learning (2021, background): “This motivates a plethora of work investigating weakly-supervised [73, 55, 41, 42, 47], semisupervised [86, 87, 84] and self-supervised learning [3, 96, 14, 4, 22, 48, 90].”