WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling

By Hao Zhang, Bo Chen, Dandan Guo, Mingyuan Zhou. 2018

130 citations recorded by the source corpus.

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sunabha Chatterjee
2. Piyush Rai
3. Il-Chul Moon
4. Wonsung Lee
5. Abhishek Kumar
6. Weonyoung Joo
7. Sungrae Park
8. Eric Eaton

In Computer Vision and Pattern Recognition

1. Sunabha Chatterjee
2. Piyush Rai
3. Il-Chul Moon
4. Wonsung Lee
5. Abhishek Kumar
6. Weonyoung Joo
7. Sungrae Park
8. Luc Van Gool

In General Social Sciences

1. Il-Chul Moon
2. Wonsung Lee
3. Weonyoung Joo
4. Sungrae Park

Citations in the corpus

• Bayesian Structure Adaptation for Continual Learning (2019, background): “Also, since gamma distribution does not have an analytic closed form KL divergence, the Weibull distribution can be a suitable alternative (Zhang et al., 2018).”
• Dirichlet Variational Autoencoder (2019, methodology): “Because of the similar PDFs between the Weibull distribution and the Gamma distribution, some prior works used the Weibull distribution as a posterior distribution of the prior Gamma distribution (Zhang et al., 2018): where k, λ > 0 .”