Unsupervised Deep Embedding for Clustering Analysis

By Junyuan Xie, Ross B. Girshick, Ali Farhadi. 2015

3,503 citations recorded by the source corpus.

Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms. Relatively little work has focused on learning representations for clustering. In this paper, we propose Deep Embedded Clustering (DEC), a method that simultaneously learns feature representations and cluster assignments using deep neural networks. DEC learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Emerging Properties in Self-Supervised Vision Transformers (2021, neutral): “Several variants allow automatic grouping of instances in the form of clustering [2, 7, 8, 21, 29, 35, 61, 65, 69].”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “Other approaches include deep volume-preserving maps (Dinh et al., 2014, 2016), deep clustering (Xie et al., 2016; Chang et al., 2017), noise as targets (NAT, Bojanowski & Joulin, 2017), self-supervised methods (Doersch & Zisserman, 2017), and co-learning (Dosovitskiy et al., 2016; Sajjadi et al.,…”
Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “DAC [8], JULE [52], DeepCluster [7], ADC [24] and DEC [51] rely on the inherent visual consistency and disentangling properties [23] of CNNs to produce cluster assignments, which are processed and reinforced in each iteration.”
Deep Clustering for Unsupervised Learning of Visual Features (2018, background): “Other clustering losses [3, 16, 35, 66, 68] have been considered to jointly learn convnet features and image clusters but they have never been tested on a scale to allow a thorough study on modern convnet architectures.”
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (2020, neutral): “Our work is also related to clustering-based methods [2, 3, 6, 7, 17, 25, 50, 53, 54, 59].”
Unsupervised Learning by Predicting Noise (2017, background): “Some of them are based on a clustering loss (Xie et al., 2016; Yang et al., 2016; Liao et al., 2016), but they are not tested at a scale comparable to that of supervised convnet training.”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Approaches in [14, 15, 16, 17, 18, 19] jointly learn image embeddings and cluster assignments, but they have not shown the ability to learn transferable representations from a large scale of images.”
Novel Class Discovery without Forgetting (2022, methodology): “DTC [20] formalized the problem of Novel Class Discovery and introduced a method based on Deep Embedded Clustering [55] for NCD, by pre-training it on the labeled data followed by learning-based clustering.”
Class-incremental Novel Class Discovery (2022, methodology): “In details, ResTune [29] uses knowledge distillation [27] on the network logits to prevent forgetting on the base classes and a clustering objective [33] with task specific network weights for the novel classes.”
Learning Discrete Representations via Information Maximizing Self-Augmented Training (2017, methodology): “Following Xie et al. (2016), we set the number of clusters to the number of ground-truth categories and evaluated clustering performance with unsupervised clustering accuracy (ACC): ACC = max m ∑N n=1 1{ln = m(cn)} N , (15) where ln and cn are the ground-truth label and cluster assignment produced…”
VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (2021, methodology): “Instead of viewing each sample as its own class, clustering-based methods [3, 46, 44, 27, 51, 6, 1, 45, 18, 5, 7] regroup them into clusters based on some similarity measure.”
Semantic Instance Segmentation with a Discriminative Loss Function (2017, background): “[41] propose a clustering objective for unsupervised learning.”
Patient Clustering Improves Efficiency of Federated Machine Learning to predict mortality and hospital stay time using distributed Electronic Medical Records (2019, methodology): “To tackle this non-IID challenge and inspired by deep embedding clustering [30], we proposed a community-based federated learning (CBFL) algorithm that clustered EMR data into several communities and simultaneously trained one model per community, so that the learning process became markedly more ef”
Unsupervised Deep Learning by Neighbourhood Discovery (2019, methodology): “Existing unsupervised deep learning methods generally fall into four different categories: (1) Clustering analysis (Caron et al., 2018; Xie et al., 2016; Yang et al., 2017), (2) Sample speciﬁcity learning (Wu et al., 2018; Bojanowski & Joulin, 2017), (3) Self-supervised learning (Doersch et al.,…”
Meta-Learning to Cluster (2019, background): “Yang et al. [2017], for instance, propose to optimize over the k-means loss, thus encouraging learning k-means friendly feature representations.”
Structural Deep Clustering Network (2020, methodology): “ods. •K-means [5]: A classical clustering method based on the raw data. •AE [7]: It is a two-stage deep clustering algorithm which performs K-means on the representations learned by autoencoder. •DEC [26]: It is a deep clustering method which designs a clustering objective to guide the learning of t”
Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision (2019, background): “There have also been recent attempts at neural-based clustering [31, 32, 33].”
Deep Bayesian Unsupervised Lifelong Learning (2021, methodology): “We report the best accuracy for DBULL since DEC [25] and VaDE [26] only report their best accuracy.”
Learning to Detect and Retrieve Objects From Unlabeled Videos (2019, methodology): “…clustering (DEC) [17] that uses the weighted student’s t-distribution as a similarity measure: Here, indices i and j are associated with the sample and cluster respectively, z i corresponds to the region embedding and μ j is the cluster centroid (for more details on DEC please refer to [17]).”