Sparse stochastic inference for latent Dirichlet allocation

By David Mimno, Matthew D. Hoffman, David M. Blei. 2012

159 citations recorded by the source corpus.

We present a hybrid algorithm for Bayesian topic models that combines the efficiency of sparse Gibbs sampling with the scalability of online stochastic inference. We used our algorithm to analyze a corpus of 1.2 million books (33 billion words) with thousands of topics. Our approach reduces the bias of variational inference and generalizes to many Bayesian hidden-variable models.

Subfields: Artificial Intelligence, General Social Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Variational Inference: A Review for Statisticians (2016, methodology): “Hoffman, Blei, and Mimno (2012) and Hoffman and Blei (2015) studied MCMC as a method of approximating coordinate updates, for example, to include structure in the variational family.”
Provable Bayesian Inference via Particle Mirror Descent (2015, methodology): “Since we evaluate their performances in terms of perplexity, which is integral over posterior, we do not need to recover the posterior, and therefore, we follow the same setting in (Ahmed et al., 2012; Mimno et al., 2012), where one particle is used in SMC and PMD to save the cost.”
Stochastic variational inference (2012, methodology): “Figure 15: 100-topic LDA inference: Holding the learning rate κ ﬁxed at 0.9, we varied the batch size. Bigger batch sizes are preferred. from the data and adjusting the noisy gradient accordingly. In Mimno et al. (2012), we developed a variant of stochastic inference that combines MCMC for the local”
Efficient mini-batch training for stochastic optimization (2014, methodology): “It has been successfully applied to large scale natural language processing [11], deep learning [7], matrix factorization [10], image classification [17], and latent variable models [22].”
Stochastic Gradient Riemannian Langevin Dynamics on the Probability Simplex (2013, methodology): “For OVB the document updates are fixed point iterations of q(zd) while for HSVG and SGRLD they are Gibbs updates of zd, the first half of which were discarded as burn-in.”
Applications of Topic Models (2017, background): “, 2014] or compute variational parameters [Mimno et al., 2012].”
Structured Stochastic Variational Inference (2014, methodology): “We also evaluated the stochastic gradient Riemannian Langevin dynamics (SGRLD) algorithm for LDA, which Patterson and Teh (2013) found outperformed the Gibbswithin-SVI method of Mimno et al. (2012). We experimented with various hyperparameter settings, including”
Model-Based Purchase Predictions for Large Assortments (2016, methodology): “This model has been used to analyze very large text corpora (Ramage et al., 2010; Mimno et al., 2012), showing that the LDA model provides the scalability needed for purchase predictions for a typical online retailer.”
Learning Deep Latent Gaussian Models with Markov Chain Monte Carlo (2017, methodology): “The idea of using MCMC to improve variational approximations has come up a number of times (e.g., De Freitas et al., 2001; Mimno et al., 2012).”