Expectation-Propogation for the Generative Aspect Model

By Tom Minka, John D. Lafferty. Conference on Uncertainty in Artificial Intelligence, 2002

574 citations recorded by the source corpus.

The generative aspect model is an extension of the multinomial model for text that allows word probabilities to vary stochastically across documents. Previous results with aspect models have been promising, but hindered by the computational difficulty of carrying out inference and learning. This paper demonstrates that the simple variational methods of Blei et al. (2001) can lead to inaccurate inferences and biased learning for the generative aspect model. We develop an alternative approach that leads to higher accuracy at comparable cost. An extension of Expectation-Propagation is used for inference and then embedded in an EM algorithm for learning. Experimental results are presented for both synthetic and real data sets.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/45b4dde8e0945912a39666f2715cdf10a4445b1c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Stochastic variational inference (2012, methodology): “Researchers have developed many methods, including Markov chain Monte Carlo methods (Grifﬁths and Steyvers, 2004), expectation propagation (Minka and Lafferty, 2002), and variational inference (Blei et al., 2003; Teh et al., 2006b; Asuncion et al., 2009).”
The Author-Topic Model for Authors and Documents (2004, background): “Then, as in the topic model, a topic is chosen from a distribution over topics speci(cid:12)c that author, and the word is generated from the chosen topic.”
Online Algorithms for Sum-Product Networks with Continuous Variables (2016, methodology): “This simpler distribution is chosen to match a subset of the moments of the exact intractable posterior, hence the name Bayesian moment matching, which is a form of assumed density filtering (Minka and Lafferty, 2002).”
WarpLDA: a Simple and Efficient O(1) Algorithm for Latent Dirichlet Allocation (2015, background): “WarpLDA is 5-15x faster than state-of-the-art LDA samplers, implying less cost of time and money.”
A Bayesian hierarchical model for learning natural scene categories (2005, methodology): “We can do this by ﬁnding the maximum likelihood estimates with expected sufﬁcient statistics computed in the E-step [1, 8].”
Collaborative Gaussian Processes for Preference Learning (2012, methodology): “Approximate inference in our model is implemented using a combination of expectation propagation (EP) [13] and variational Bayes (VB) [7].”
Topic and Role Discovery in Social Networks (2005, unknown): “(Blei et al., 2003), Gibbs sampling (Griths & Steyvers, 2004; Steyvers et al., 2004; Rosen-Zvi, Griths, Steyvers, & Smyth, 2004), and expectation propagation (Griths & Steyvers, 2004; Minka & Laerty, 2002 ).”
Modeling online reviews with multi-grain topic models (2008, background): “Unfortunately exact inference in such model is intractable and various approximations have been considered [3, 23, 14].”
A classification for community discovery methods in complex networks (2011, methodology): “A number of approximate inference algorithms for mixed membership models have recently appeared such as mean-field variational methods [148], expectation propagation [149] and Monte Carlo Markov chain sampling [150].”
A Model of Text for Experimentation in the Social Sciences (2016, background): “Perhaps the simplest topic model, to which, arguably, much of the recent interest in statistical text analysis research can be ascribed, is known as the latent Dirichlet allocation (LDA henceforth), or also as the generative aspect model (Blei, Ng, and Jordan 2001, 2003; Minka and Lafferty 2002).”
Topical N-Grams: Phrase and Topic Discovery, with an Application to Information Retrieval (2007, methodology): “Approximate inference techniques such as variational methods [12], Gibbs sampling [1] and expectation propagation [17] have been developed to address this issue.”
An overview of topic modeling and its current applications in bioinformatics (2016, methodology): “Classic approaches to an inference algorithm in LDA are expectation-propagation (EP) (Minka and Lafferty 2002), collapsed Gibbs sampling (Griffiths and Steyvers 2004), and variational Bayesian inference (VB) (Blei et al. 2003).”
Streaming Variational Bayes (2013, background): “We see that, as a result, D must be speciﬁed in advance, appears in each step of SVI (see Alg.”
Bayesian Gaussian Process Classification with the EM-EP Algorithm (2006, methodology): “It has been shown that EP provides better accuracy than variational methods at a comparable cost for the generative aspect model [21].”
Expectation Propagation as a Way of Life: A Framework for Bayesian Inference on Partitioned Data (2014, background): “In particular, running the EP updates in parallel often yields a deviated global approximation when compared to the result obtained with sequential updates (Minka and Lafferty, 2002; Jylänki et al., 2011).”