Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization

By Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong (Tom) Cai, Heng Huang. IEEE International Conference on Computer Vision, 2017

580 citations recorded by the source corpus.

In this paper, we propose a new clustering model, called DEeP Embedded Regularized ClusTering (DEPICT), which efficiently maps data into a discriminative embedding subspace and precisely predicts cluster assignments. DEPICT generally consists of a multinomial logistic regression function stacked on top of a multi-layer convolutional autoencoder. We define a clustering objective function using relative entropy (KL divergence) minimization, regularized by a prior for the frequency of cluster assignments. An alternating strategy is then derived to optimize the objective by updating parameters and estimating cluster assignments. Furthermore, we employ the reconstruction loss functions in our autoencoder, as a data-dependent regularization term, to prevent the deep embedding function from overfitting. In order to benefit from end-to-end optimization and eliminate the necessity for layer-wise pre-training, we introduce a joint learning framework to minimize the unified clustering and reconstruction loss functions together and train all network layers simultaneously. Experimental results indicate the superiority and faster running time of DEPICT in real-world clustering tasks, where no labeled data is available for hyper-parameter tuning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d9ae299d42b418467655d5e5c40881f70308e2d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MixNMatch: Multifactor Disentanglement and Encoding for Conditional Image Generation (2019, methodology): “JULE and DEPICT focus more on background and rough shape information instead of fine grained details, and thus have relatively low performance.”
Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation (2021, background): “Unsupervised clustering has been studied for decades with many classical approaches [35, 3, 9] and deep learning based solutions [50, 16, 40] being proposed.”
Differentially Private Mixture of Generative Neural Networks (2017, methodology): “ural network with a clustering algorithm, and use the internal representation provided by the autoencoder, i.e., the encoder output, as features for clustering. A different training method is used in [17,31,54], where autoencoders are initially pre-trained, and then ﬁne-tuned using the cluster assig”
OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in an Open World (2020, methodology): “Many classic methods [15, 30] and deep learning methods [5, 27, 28] have been proposed.”
A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture (2018, methodology): “Additionally, noise can also be added to the inputs of each layer [14].”
Deep Comprehensive Correlation Mining for Image Clustering (2019, methodology): “[17] also uses the KLdivergence loss, but adds a noisy encoder to learn more robust representation.”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, methodology): “Dizaji [10] proposes DEPICT, a method that trains a convolutional auto-encoder with a softmax layer stacked on-top of the encoder.”
FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery (2018, methodology): “…work on unsupervised object category discovery focus only on clustering entry-level categories (e.g., birds vs. cars vs. dogs) [17, 44, 32, 53, 49, 15], while existing work on ﬁne-grained recognition focus exclusively on the supervised setting in which ground-truth ﬁne-grained category…”
SpectralNet: Spectral Clustering using Deep Neural Networks (2018, methodology): “Unfortunately, we were unable to ﬁnd a setting that will yield an appropriate clustering on any of the datasets for DCN, VaDE and DEPICT.”
Deep Learning for Anomaly Detection (2020, background): “n feature representations tailored for a specific clustering algorithm, is the most critical component of this anomaly detection method. A number of studies have explored this problem in recent years [25, 37, 48, 152, 163, 166, 167]. The main motivation is due to the fact that the performance of clu”
Associative Deep Clustering: Training a Classification Network with No Labels (2018, methodology): “Deep Embedded Regularized Clustering (DEPICT) [5] is similar to DEC in that feature representations and cluster assignments are simultaneously learned using a deep network.”
Contrastive Clustering (2021, methodology): “To solve the problem, deep clustering [10], [13] utilizes neural networks to extract representative information from images for facilitating the downstream clustering tasks.”
Neighborhood Contrastive Learning for Novel Class Discovery (2021, unknown): “To this end, many shallow [1, 23, 34] and deep learning based methods [6, 9, 24, 29, 32, 33, 36] have been proposed.”
Unsupervised Deep Learning by Neighbourhood Discovery (2019, background): “With the surge of deep learning techniques, recent studies have attempted to optimise clustering analysis and representation learning jointly for maximising their complementary beneﬁts (Caron et al., 2018; Xie et al., 2016; Yang et al., 2017; Dizaji et al., 2017).”
Learning to Discover Novel Visual Categories via Deep Transfer Clustering (2019, background): “Recently, there have been more and more works on clustering in the deep learning literature (e.g., [6, 12, 38, 39, 40]).”