Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering

By Bo Yang, Xiao Fu, N. Sidiropoulos, Mingyi Hong. International Conference on Machine Learning, 2016

990 citations recorded by the source corpus.

Most learning approaches treat dimensionality reduction (DR) and clustering separately (i.e., sequentially), but recent research has shown that optimizing the two tasks jointly can substantially improve the performance of both. The premise behind the latter genre is that the data samples are obtained via linear transformation of latent representations that are easy to cluster; but in practice, the transformation from the latent space to the data can be more complicated. In this work, we assume that this transformation is an unknown and possibly nonlinear function. To recover the 'clustering-friendly' latent representations and to better cluster the data, we propose a joint DR and K-means clustering approach in which DR is accomplished via learning a deep neural network (DNN). The motivation is to keep the advantages of jointly optimizing the two tasks, while exploiting the deep neural network's ability to approximate any nonlinear function. This way, the proposed approach can work well for a broad class of generative models. Towards this end, we carefully design the DNN structure and the associated joint optimization criterion, and propose an effective and scalable algorithm to handle the formulated optimization problem. Experiments using different real datasets are employed to showcase the effectiveness of the proposed approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ca9f84c3922004ec6133aa9c2048ceeb17702fee

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Clustering with Convolutional Autoencoders (2017, methodology): “The most widely used neural networks in deep clustering algorithms are Stacked AutoEncoders (SAE) [12,13,16,18].”
Unsupervised Deep Learning by Neighbourhood Discovery (2019, methodology): “…unsupervised deep learning methods generally fall into four different categories: (1) Clustering analysis (Caron et al., 2018; Xie et al., 2016; Yang et al., 2017), (2) Sample speciﬁcity learning (Wu et al., 2018; Bojanowski & Joulin, 2017), (3) Self-supervised learning (Doersch et al., 2015;…”
Deep Amortized Clustering (2019, methodology): “Deep clustering methods There is a growing interest in developing clustering methods using deep networks for complex data (Yang et al., 2016a;b; Xie et al., 2015; Li et al., 2017; Ji et al., 2018).”
Differentially Private Mixture of Generative Neural Networks (2017, background): “Some proposals [25,56,60] jointly train an autoencoder neural network with a clustering algorithm, and use the internal representation provided by the autoencoder, i.”
Classification-Based Anomaly Detection for General Data (2020, methodology): “Recently, deep learning methods (autoencoders or variational autoencoders) were used to learn deep features which are sometimes easier to model than raw features (Yang et al., 2017).”
Prototypical Contrastive Learning of Unsupervised Representations (2020, background): “Approaches in [14, 15, 16, 17, 18, 19] jointly learn image embeddings and cluster assignments, but they have not shown the ability to learn transferable representations from a large scale of images.”
Associative Deep Clustering: Training a Classification Network with No Labels (2018, methodology): “Other methods avoid trivial representations by using additional techniques unrelated to clustering, such as autoencoder reconstruction loss during pre-training or entire training [20, 41, 42, 26].”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, methodology): “And then, a K -means loss is adopted at the clustering phase to train a fully connected autoencoder network [42], which is a joint approach of dimensionality reduction and K -means clustering.”
Deep Graph Clustering via Dual Correlation Reduction (2021, methodology): “…in the trivial constant representation; 3) our method improves the auto-encoder-based clustering methods, i.e., AE (Yang et al. 2017), DEC (Yang et al. 2017) and IDEC (Guo et al. 2017), by a large margin, all of which have been veriﬁed strong representation learning capacity for…”
Clustering with Deep Learning: Taxonomy and New Methods (2018, methodology): “Such a transformation could be beneﬁcial for the clustering sometimes, but using a clustering loss usually yields better results (Xie et al., 2016; Yang et al., 2016a).”
A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture (2018, methodology): “• Deep Clustering Network (DCN): DCN [13] is one of the most remarkable methods in this field, which combines autoencoder with the k-means”
Learning to Discover Novel Visual Categories via Deep Transfer Clustering (2019, background): “Recently, there have been more and more works on clustering in the deep learning literature (e.g., [6, 12, 38, 39, 40]).”
ClusterGAN : Latent Space Clustering in Generative Adversarial Networks (2018, methodology): “Since DCN has been shown to outperform various deep-learning based clustering algorithms, we reported its metrics from the paper (Yang et al. 2017) for MNIST and Pendigits.”
OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in an Open World (2020, methodology): “Many classic methods [15, 30] and deep learning methods [5, 27, 28] have been proposed.”
SpectralNet: Spectral Clustering using Deep Neural Networks (2018, methodology): “Unfortunately, we were unable to ﬁnd a setting that will yield an appropriate clustering on any of the datasets for DCN, VaDE and DEPICT.”