Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering

By Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, Hanning Zhou. 2016

869 citations recorded by the source corpus.

Clustering is among the most fundamental tasks in machine learning and artificial intelligence. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into an observable. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using the Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 5 benchmarks from various modalities. Moreover, by VaDE's generative nature, we show its capability of generating highly realistic samples for any specified cluster, without using supervised information during training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Continual Unsupervised Representation Learning (2019, methodology): “…the underlying model to learn in a more traditional setting with the entire dataset shufﬂed, and compare with existing work in clustering and representation learning: the VAE (Kingma & Welling, 2013), DirichletVAE (Joo et al., 2019), SBVAE (Nalisnick & Smyth, 2017), and VaDE (Jiang et al., 2017).”
Lifelong Generative Modeling (2017, background): “These models have wide ranging applications such as clustering [77, 52, 85] and pre-training [68, 95].”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Using a mixture distribution for latent variables in a VAE was explored in Dilokthanakul et al. (2016); Nalisnick et al. (2016), and Jiang et al. (2017) for unconditional image generation and text topic modeling.”
A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture (2018, background): “• Variational Deep Embedding (VaDE): VaDE [15] consider the generative model p(x, z, c) = p(x|z)p(z|c)p(c),In this model, an observed sample x is generated by the following process:”
Structural Deep Clustering Network (2020, methodology): “For example, [27] uses the representation learned by autoencoder in K-means; [4, 26] leverage a clustering loss to help autoencoder learn the data representation with high cluster cohesion [22], and [9] uses a variational autoencoder to learn better data representation for clustering.”
Dirichlet Variational Autoencoder (2019, background): “nces in most cases. 1 INTRODUCTION A Variational Autoencoder (VAE) (Kingma & Welling, 2014c) brought success in deep generative models (DGMs) with a Gaussian distribution as a prior distribution (Jiang et al., 2017; Miao et al., 2016; 2017; Srivastava & Sutton, 2017). If we focus on the VAE,”
NAT: Neural Architecture Transformer for Accurate and Compact Architectures (2019, background): “…networks (DNNs) [25] have been producing state-of-the-art results in many challenging tasks including image classiﬁcation [12, 23, 43, 60, 61, 18, 11, 56], face recognition [39, 44, 59], brain signal processing [34, 35], video analysis [48, 53, 52] and many other areas [58, 57, 24, 3, 10, 9,…”
Deep Generative Models in Engineering Design: A Review (2021, methodology): “pose an approach leveraging a Variational Deep Embedding (VaDE) [125].”
Learning Disentangled Representations for Recommendation (2019, background): “Though a few existing approaches [14, 10, 7, 13, 24] do notice that a dataset can contain samples from different concepts, i.”
Hyperprior Induced Unsupervised Disentanglement of Latent Representations (2018, background): “In addition, the diagonal covariance I indicates a prior expectation that underlying data representation comprises statistically independent Gaussians (one for each latent dimension), and is therefore disentangled.”
Deep Bayesian Unsupervised Lifelong Learning (2021, methodology): “We report the best accuracy for DBULL since DEC [25] and VaDE [26] only report their best accuracy.”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, methodology): “VaDE [18] is a variational autoencoder method for deep embedding, and combines a Gaussian Mixture Model for clusering.”