Deep Clustering for Unsupervised Learning of Visual Features

By Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze. 2018

2,342 citations recorded by the source corpus.

Clustering is a class of unsupervised learning methods that has been extensively applied and studied in computer vision. Little work has been done to adapt it to the end-to-end training of visual features on large-scale datasets. In this work, we present DeepCluster, a clustering method that jointly learns the parameters of a neural network and the cluster assignments of the resulting features. DeepCluster iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. We apply DeepCluster to the unsupervised training of convolutional neural networks on large datasets like ImageNet and YFCC100M. The resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Representation & Self-Supervised Learning

Citations in the corpus

Are we done with ImageNet? (2020, neutral): “Similarly, successful representation learning techniques applied to ImageNet [9, 10, 11] have yielded corresponding gains elsewhere [12, 13, 14, 15].”
Self-Supervised Learning of Video-Induced Visual Invariances (2019, background): “Prior work has explored diverse self-supervision cues such as predicting the spatialcontext [12], colorization [71], equivariance to transformations [18, 44]; alongside unsupervised techniques such as clustering [6, 72], generative modelling [14, 32], and exemplar learning [15].”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “Our method, which learns only from Ima-geNet data using a single unsupervised objective, reaches 70.6% when equipped with a ResNet-101 feature extractor (as for most competing methods [15] but not all [8, 9]).”
Contrastive Multiview Coding (2019, methodology): “As many previous unsupervised methods are evaluated with AlexNet [43] on ImageNet [16, 42, 17, 84, 54, 18, 85, 55, 22, 11, 83], we also include the the results of CMC using this network.”
BEiT: BERT Pre-Training of Image Transformers (2021, methodology): “Another strand of methods use clustering to organize image examples (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020; Li et al., 2021).”
Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several variants allow automatic grouping of instances in the form of clustering [2, 7, 8, 21, 29, 35, 61, 65, 69].”
Exploring Simple Siamese Representation Learning (2020, methodology): “Like contrastive learning, clustering-based methods require either a memory bank [5, 6, 1], large batches [7], or a queue [7] to provide enough samples for clustering.”
SLIP: Self-supervision meets Language-Image Pre-training (2021, methodology): “As we conﬁrm again later in this work, the standard self-supervised evaluation task of ImageNet clas-siﬁcation from frozen features heavily biases results against models not also pre-trained on ImageNet [5].”
A critical analysis of self-supervision, or what we can learn from a single image (2019, methodology): “…(Donahue et al., 2017) utilizes a generative adversarial task, RotNet (Gidaris et al., 2018) exploits the photographic bias in the dataset and DeepClus-ter (Caron et al., 2018) learns stable feature representations under a number of image transformations by proxy labels obtained from clustering.”
Unsupervised Embedding Learning via Invariant and Spreading Instance Feature (2019, methodology): “Existing works can be roughly categorized into three categories [3]: 1) generative models , this approach aims at learning a parameterized mapping between images and predeﬁned noise signals, which constrains the distribution between raw data and noises [46].”
Local Aggregation for Unsupervised Learning of Visual Embeddings (2019, methodology): “DeepCluster [7] (DC) trains a DCNN in a series of iterative rounds.”
Self-Supervised GANs via Auxiliary Rotation Loss (2018, methodology): “Other surrogate tasks include image inpainting [45], predicting the color channels from a grayscale image [25], and predicting the unsupervised clustering classes [35].”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “DAC [8], JULE [52], DeepCluster [7], ADC [24] and DEC [51] rely on the inherent visual consistency and disentangling properties [23] of CNNs to produce cluster assignments, which are processed and reinforced in each iteration.”
ScaleNet: An Unsupervised Representation Learning Method for Limited Information (2023, methodology): “Image clustering-based methods, such as DeepCluster [4], SwAV [5], and SeLa [2], generate labels by an unsupervised learning method initially and then use the subsequent assignments in a supervised way to update the weights of the network.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “Our approach draws inspiration from the DeepCluster method for self-supervised visual learning [23]; however, HuBERT benefits from the masked prediction loss over speech sequences to represent their sequential structure.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “While supervised learning has been successful in a variety of computer vision tasks [17, 9, 38, 29], self-supervised representation learning from unlabeled data has attracted increasing attention in recent years [4, 27].”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, positive): “DeepCluster-v2 and SeLa-v2, which are obtained by applying various training improvements introduced in other self-supervised learning papers to DeepCluster [6] and SeLa [2].”
Prototypical Contrastive Learning of Unsupervised Representations (2020, methodology): “Closer to our work, DeepCluster [20] learns from millions of images by performing iterative clustering and unsupervised representation learning.”