On Smoothing and Inference for Topic Models

By Arthur U. Asuncion, Max Welling, Padhraic Smyth, Yee Whye Teh. Conference on Uncertainty in Artificial Intelligence, 2009

615 citations recorded by the source corpus.

Latent Dirichlet analysis, or topic modeling, is a flexible latent variable framework for modeling high-dimensional sparse count data. Various learning algorithms have been developed in recent years, including collapsed Gibbs sampling, variational inference, and maximum a posteriori estimation, and this variety motivates the need for careful empirical comparisons. In this paper, we highlight the close connections between these approaches. We find that the main differences are attributable to the amount of smoothing applied to the counts. When the hyperparameters are optimized, the differences in performance among the algorithms diminish significantly. The ability of these algorithms to achieve solutions of comparable accuracy gives us the freedom to select computationally efficient approaches. Using the insights gained from this comparative study, we show how accurate topic models can be learned in several seconds on text corpora with thousands of documents.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/239243dfa44e14b4fca15675d25cb655fb91a7b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Variational inference in nonconjugate models (2012, positive): “We follow the testing framework of Asuncion et al. (2009) and Blei and Lafferty (2007).”
Structured Stochastic Variational Inference (2014, methodology): “For example, for latent Dirichlet allocation one could adapt the CVB0 method of Asuncion et al. (2009) to use a fixed value of β, resulting in an algorithm akin to that of Foulds et al.”
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, methodology): “[3] have shown that this MAP solution is almost identical with the solution of CGS, with proper hyper-parameters.”
Reducing Wrong Labels in Distant Supervision for Relation Extraction (2012, neutral): “We apply the following approximation (Asuncion et al., 2009).”
A Model of Text for Experimentation in the Social Sciences (2016, methodology): “... Comparative Performance Evaluationwith State-of-the-Art To provide a fully automated comparison of our model to existing alternatives, we estimate the heldout likelihood using the document completion approach (Asuncion et al. 2009; Wallach et al. 2009b).”
Mining Text Data (2012, methodology): “This approach works with the use of a summarization methodology, in which we use the concept of condensed droplets [3] in order to create concise representations of the underlying clusters.”
Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models (2014, methodology): “This is a good way to evaluate probabilistic topic models (Blei and Lafferty, 2007; Asuncion et al., 2009).”
Mining meaning from online ratings and reviews: Tourist satisfaction analysis using latent dirichlet allocation (2017, methodology): “2003), Gibbs sampling methods (Griffiths and Steyvers 2004), or the maximum likelihood estimate method (Asuncion et al. 2009).”
Rethinking LDA: Why Priors Matter (2009, methodology): “[1] demonstrated that the choice of inference method has negligible effect on the probability of held-out documents or inferred topics.”
stm: An R Package for Structural Topic Models (2019, methodology): “We use the document-completion held-out likelihood method which is the estimation of the probability of words appearing within a document when those words have been removed from the document in the estimation step (Asuncion et al. 2009; Wallach et al. 2009; Hoffman et al. 2013).”
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation (2017, background): “It is important to note that both variational and samplingbased algorithms provide similarly accurate results [30].”
Online Variational Inference for the Hierarchical Dirichlet Process (2011, methodology): “This is similar to the metrics used in [3, 22], which tries to avoid comparing different hyperparameters.”
Learning Discriminative Projections for Text Similarity Measures (2011, methodology): “In the experiments in this paper, our implementation of PLSA is LDA with maximum a posteriori (MAP) inference, which was shown to be comparable to the current best Bayesian inference methods for LDA (Asuncion et al., 2009).”
Seeing Like the Fed: Culture, Cognition, and Framing in the Failure to Anticipate the Financial Crisis of 2008 (2017, background): “Perplexity5 and similar predictiveness metrics typically require division of documents into a training set and a test set (Asuncion et al. 2009).”
Model-Based Purchase Predictions for Large Assortments (2016, methodology): “This extends the text modeling implementation of LDA, where these probabilities are typically considered to be known, or at best calibrated using heuristics (Wallach et al., 2009; Asuncion et al., 2009).”
Simultaneously Discovering and Quantifying Risk Types from Textual Risk Disclosures (2014, background): “2007, Asuncion et al. 2009, Wallach et al. 2009, Zhai et al. 2012) have attempted to compare their performance. However, the findings are mixed. Following Blei and Jordan (2006), we resort to the empirical experiments for comparing the different learning algorithms in our context.”
Online Learning for Latent Dirichlet Allocation (2010, methodology): “Although the choice of approximate posterior introduces bias, VB is empirically shown to be faster than and as accurate as MCMC, which makes it an attractive option when applying Bayesian models to large datasets [1] [2] [3].”
Topic Modeling Using Latent Dirichlet allocation (2021, methodology): “Even though LSI can find topics from the large set of documents, it does not describe the document generation process.”
Beta-Negative Binomial Process and Poisson Factor Analysis (2011, methodology): “The performance measure is similar to those used in Asuncion et al. (2009); Wallach et al. (2009); Wang et al. (2011).”