Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

By Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, M. J. Lee, Hugh Salimbeni, Kai Arulkumaran, Murray Shanahan. arXiv.org, 2016

734 citations recorded by the source corpus.

We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of this heuristic and provide an intuition of the various processes with the help of visualizations. Finally, we demonstrate the performance of our model on synthetic data, MNIST and SVHN, showing that the obtained clusters are distinct, interpretable and result in achieving competitive performance on unsupervised clustering to the state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/06b26678cda5f243338f43b18b7532228273a7f0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “The VAE with the VampPrior achieves better density estimation performance than the VAE with the standard Gaussian prior and the VAE with the Gaussian mixture prior (Dilokthanakul et al. 2016).”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, methodology): “…ERD [17] and acLSTM [45]; (2) Stochastic motion prediction methods, including CVAE based methods, Pose-Knows [68] and MT-VAE [71], as well as a CGAN based method, HP-GAN [6]; (3) Diversity-promoting methods for generative models, including Best-of-Many [8], GMVAE [15], DeLiGAN [27], and DSF [73].”
Hyperspherical Variational Auto-Encoders (2018, methodology): “One approach to obtain a more flexible prior is to use a simple mixture of Gaussians (MoG) prior (Dilokthanakul et al., 2016).”
Auto-Encoding Total Correlation Explanation (2018, methodology): “After training the model, we ﬁnd that the learned discrete variable z (2) on the top layer gives us an unsupervised classiﬁcation accuracy of 85%, which is competitive with the more complex method shown in (Dilokthanakul et al., 2016).”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Using a mixture distribution for latent variables in a VAE was explored in Dilokthanakul et al. (2016); Nalisnick et al. (2016), and Jiang et al. (2017) for unconditional image generation and text topic modeling.”
Graph Information Bottleneck (2020, positive): “After the model computes φ , which is instantiated as follows for the two versions, To estimate XIB ( l ) , we set Q ( Z X ) as a mixture of Gaussians with learnable parameters [27].”
LaDDer: Latent Data Distribution Modelling with a Generative Prior (2020, positive): “We compare our method to 4 other approaches, including the original VAE with a normal prior [22, 35], VAE with a GMM prior [7, 18, 27], VAE with a hierarchical prior [24] where a generative prior with a normal hyper prior is used, and VampPrior [38] where the prior is modelled as an average encoding”
Learning Hierarchical Priors in VAEs (2019, methodology): “Several works improve the VAE by learning more complex priors such as the stick-breaking prior (Nalisnick & Smyth, 2017), a nested Chinese Restaurant Process prior (Goyal et al., 2017), Gaussian mixture priors (Dilokthanakul et al., 2016), or autoregressive priors (Chen et al., 2016b).”
Learning Disentangled Representations for Recommendation (2019, background): “Though a few existing approaches [14, 10, 7, 13, 24] do notice that a dataset can contain samples from different concepts, i.”
A Survey of Clustering With Deep Learning: From the Perspective of Network Architecture (2018, background): “In this model, an observed sample x is generated as the following process: c ∼ Cat(1/K ),n ∼ N (0, I ) z ∼ N (µc(n), σ 2c(n)) x ∼ N (µx(z), σ x(z)I ) or B(µx(z)) VOLUME 6, 2018 39509 Note that GMVAE is a little complex than VaDE and has worse results empirically.”
Gaussian mixture models with Wasserstein distance (2018, methodology): “For example, in [5] a Gaussian-mixture latent-variable model (GM-LVM) was studied, and the authors were unable to train their model on MNIST using variational inference without substantially modifying the VAE objective.”
Deep generative models for T cell receptor protein sequences (2019, background): “However, VAE variants do use other distributions in place of normal (Dilokthanakul et al., 2016; Davidson et al., 2018).”
Variational Memory Addressing in Generative Models (2017, positive): “A number of architectures for VAEs augmented with mixture priors have been proposed, but they do not use the mixture component indicator variable to index memory and integrate out the variable instead [7, 22], which prevents them from scaling to a large number of mixing components.”
Deep Learning for Anomaly Detection (2020, background): “n feature representations tailored for a specific clustering algorithm, is the most critical component of this anomaly detection method. A number of studies have explored this problem in recent years [25, 37, 48, 152, 163, 166, 167]. The main motivation is due to the fact that the performance of clu”
Resampled Priors for Variational Autoencoders (2018, methodology): “[5] Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville.”
Deep Spectral Clustering Using Dual Autoencoder Network (2019, background): “In addition, Gaussian Mixture Variational Autoencoder (GMVAE) [9] shows that minimum information constraint can be utilized to mitigate the effect of over-regularization in VAEs and provides an unsupervised clustering within the VAE framework considering a Gaussian mixture as a prior distribution.”
Every Smile is Unique: Landmark-Guided Diverse Smile Generation (2018, methodology): “The second phase consists on fine-tuning the VAE and training the first layer LSTM on the dataset of sequences of landmark images {yn}Nn=1 for 20 epochs.”