Stick-Breaking Variational Autoencoders

By Marco Cote. 2016

178 citations recorded by the source corpus.

We extend Stochastic Gradient Variational Bayes to perform posterior inference for the weights of Stick-Breaking processes. This development allows us to define a Stick-Breaking Variational Autoencoder (SB-VAE), a Bayesian nonparametric version of the variational autoencoder that has a latent representation with stochastic dimensionality. We experimentally demonstrate that the SB-VAE, and a semi-supervised variant, learn highly discriminative latent representations that often outperform the Gaussian VAE's.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Lastly, KL-divergence between Kuma and Beta distribution can be approximated in closed form [70].”
Bayesian Structure Adaptation for Continual Learning (2019, methodology): “For Beta distributed v ’s, the Kumaraswamy Reparameterization Gradient technique (Nalisnick & Smyth, 2016) is used.”
Continual Unsupervised Representation Learning (2019, methodology): “As a simple non-parametric approach, it serves this purpose, and has also been used extensively in previous evaluations as a result (Nalisnick & Smyth, 2017; Joo et al., 2019).”
Lifelong Generative Modeling (2017, background): “These models have wide ranging applications such as clustering [77, 52, 85] and pre-training [68, 95].”
VAE with a VampPrior (2017, methodology): “In [30] a Dirichlet process prior using a stickbreaking process was proposed, while [14] proposed a nested Chinese Restaurant Process.”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, methodology): “We can use this fact to reparame-terize expectations (Nalisnick and Smyth, 2016).”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “For example, non-parametric Bayesian distribution (Nalisnick and Smyth 2017) and hy-perspherical distribution (Davidson et al. 2018) are used for the prior.”
Dirichlet Variational Autoencoder (2019, methodology): “Fundamentally, we applied the same experimental settings to GVAE, SBVAE and DirVAE in this experiment, as speciﬁed by the authors in Nalisnick & Smyth (2017).”
Hierarchical Indian buffet neural networks for Bayesian continual learning (2019, methodology): “Stick-breaking probabilities have been placed directly as the VAE latent state [Nalisnick and Smyth, 2017].”
Differentiable Causal Discovery Under Latent Interventions (2022, background): “The work by Nalisnick and Smyth (2017) introduces a VAE where the stick-breaking weights are the latent variables.”
Hyperprior Induced Unsupervised Disentanglement of Latent Representations (2018, background): “2016), Dirichlet process (Nalisnick and Smyth 2016), Beta, Gamma, and von Mises (Figurnov, Mohamed, and Mnih 2018).”
Explorations in Homeomorphic Variational Auto-Encoding (2018, methodology): “Nalisnick & Smyth (2017) introduce a non-parametric prior applying a truncated stick-breaking method.”
Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering (2016, methodology): “SB-VAE Nalisnick and Smyth [2016] also applied Bayesian nonparametric techniques on VAE, which derived a stochastic latent dimensionality by a stick-breaking prior and achieved good performance on semi-supervised classiﬁcation tasks.”
PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior (2021, background): “…distribution for deep generative model are not new, and there has been a variety of studies investigating a better prior, ranging from hand-crafted (Nalisnick & Smyth, 2017; Tomczak & Welling, 2018), autoregressive (Chen et al., 2017), vector quantization (Razavi et al., 2019), prior encoder…”
Variational Russian Roulette for Deep Bayesian Nonparametrics (2019, methodology): “Similarly, recent work on amortized inference in Bayesian non-parametrics relies on truncation, albeit sometimes within an outer loop that searches over the truncation size (Miao et al., 2017; Nalisnick & Smyth, 2017; Chatzis, 2014; Singh et al., 2017).”
Indian Buffet Neural Networks for Continual Learning (2019, methodology): “This reparameterisation has been used successfully to learn a discrete latent hidden representation in a VAE where the parameters a and b are learnt using stochastic gradient descent [19, 20].”
PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior (2021, result): “…an informative prior distribution for deep generative model are not new, and there has been a variety of studies investigating a better prior, ranging from hand-crafted [21, 38], autoregressive [3], vector quantization [27], prior encoder [30], and data-dependent approaches similar to ours [19].”
Nonparametric Bayesian Deep Networks with Local Competition (2018, methodology): “To address this issue, one can approximate the Beta( a k , b k ) - distributed stick variables, u k , via the Kumaraswamy distribution (Kumaraswamy, 1980): Samples from this distribution can be reparameterized as follows (Nalisnick & Smyth, 2016):”
Hyperspherical Variational Auto-Encoders (2018, methodology): “A non-parametric prior is proposed in Nalisnick and Smyth (2017), utilizing a truncated stick-breaking process.”