Efficient methods for topic model inference on streaming document collections

By Limin Yao, David Mimno, Andrew McCallum. Knowledge Discovery and Data Mining, 2009

464 citations recorded by the source corpus.

Topic models provide a powerful tool for analyzing large text collections by representing high dimensional data in a low dimensional subspace. Fitting a topic model given a set of training documents requires approximate inference techniques that are computationally expensive. With today's large-scale, constantly expanding document collections, it is useful to be able to infer topic distributions for new documents without retraining the model. In this paper, we empirically evaluate the performance of several methods for topic inference in previously unseen documents, including methods based on Gibbs sampling, variational inference, and a new method inspired by text classification. The classification-based inference method produces results similar to iterative inference methods, but requires only a single matrix multiplication. In addition to these inference methods, we present SparseLDA, an algorithm and data structure for evaluating Gibbs sampling distributions. Empirical results indicate that SparseLDA can be approximately 20 times faster than traditional LDA and provide twice the speedup of previously published fast sampling methods, while also using substantially less memory.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/60707929c23c159b0bf607d3a245e3f834980d19

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

Citations in the corpus

Unsupervised Modeling of Twitter Conversations (2010, methodology): “Note that our implementations can likely scale to larger data by using techniques such as SparseLDA (Yao et al., 2009).”
WarpLDA: a Simple and Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, background): “With WarpLDA users can learn up to one million topics from hundreds of millions of documents in a few hours, at the speed of 2G tokens per second, or learn topics from small-scale datasets in seconds.”
Open domain event extraction from twitter (2012, methodology): “Prediction for new data is performed using a streaming approach to inference [56].”
High-Performance Distributed ML at Scale through Parameter Server Consistency Models (2014, background): “…14], or special-purpose solvers tailored to specific categories of ML applications: CCD++ [18] for Matrix Factorization, Fugue [9] for constrained MF, Vowpal Wabbit for regression/classification problems via stochastic optimization [10], and Yahoo LDA as well as Google plda for topic modeling [17].”
Streaming First Story Detection with application to Twitter (2010, background): “, 2009), and topic modelling on streaming document collections (Yao et al., 2009).”
An architecture for parallel topic models (2010, background): “Moreover, collapsed sampling [10, 9, 7] tends to lead to somewhat better models than a variational approach.”
Mining Text Data (2012, methodology): “For example, the work in [72] uses an EM-approach in order to perform supervised clustering (and classification) of the documents, when a mixture of labeled and unlabeled data is available.”
WarpLDA: a Cache Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, methodology): “Let ᾱ = ∑K k=1 αk and β̄ = V β.”
Petuum: A New Platform for Distributed Machine Learning on Big Data (2013, methodology): “Our experiments use the SparseLDA algorithm [30], to ensure a fair comparison with YahooLDA [28] (which also uses SparseLDA).”
Applications of Topic Models (2017, methodology): “In this survey, we focus on Gibbs sampling, which is simple, intuitive, and—with some clever tricks specific to topic models—fast [Yao et al., 2009].”
Social big data: Recent achievements and new challenges (2015, methodology): “However, new, efficient, arallel implementations of this algorithm have appeared [66] in atempts to improve its efficiency.”
Latent Variable Models of Selectional Preference (2010, background): “Yao et al. (2009) propose a reformulation that removes this bottleneck by separating the probability mass p(z|n, v) into a number of buckets, some of which only require iterating over the topics currently assigned to instances of type n, typically far fewer than the total number of topics.”
Interactive Topic Modeling (2011, background): “After the constraint was added, which encouraged Russian and Soviet terms to be in the same topic, non-Russian terms gained increased prominence in Topic 1, and “Moscow” (which was not part of the constraint) appeared in Topic 20. ﬁer (Hall et al., 2009).”
Topic Modeling Using Latent Dirichlet allocation (2021, background): “8 It proved itself 20 times quicker than plain LDA [166].”
Multiple domain user personalization (2011, background): “[20] describes an equivalent sampler which is significantly faster through careful choice of data structures and creative rearrangement of the conditional probabilities.”
Gaussian LDA for Topic Models with Word Embeddings (2015, methodology): “Therefore we implement the online version of our algorithm using Gibbs sampling following (Yao et al., 2009).”
Strategies and Principles of Distributed Machine Learning on Big Data (2015, background): “In the interest of keeping this article focused, we refer the reader to [61] for an appropriate introduction.”