Presenter : HMW Category : graphical models Preference : Oral Evaluation Methods for Topic Models

By Hanna M. Wallach, Iain Murray, Ruslan Salakhutdinov, David Mimno. 2009

Subfields: Natural Language Processing, Evaluation & Benchmarks, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

In Probabilistic & Bayesian Methods

Citations in the corpus

Rethinking LDA: Moment Matching for Discrete ICA (2015, positive): “Given that we are in an unsupervised setting and that moreover the evaluation of LDA is a difficult task [16], setting this parameter is non-trivial.”
Hierarchical Variational Models (2015, background): “This is a document complete evaluation metric (Wallach et al., 2009) where the words are tested independently.”
Deep Exponential Families (2014, methodology): “This is similar to the document completion evaluation metric [38] except we query the test words independently.”
Sparse stochastic inference for latent Dirichlet allocation (2012, methodology): “We evaluate held-out probability using the left-to-right sequential sampling method (Wallach et al., 2009; Buntine, 2009).”
Topically Driven Neural Language Model (2017, background): “There are various ways to estimate test perplexity (Wallach et al., 2009), but Chang et al. (2009) show that perplexity does not correlate with the coherence of the generated topics.”
Replicated Softmax: an Undirected Topic Model (2009, methodology): “We also find AIS attractive because it not only provides a good estimate of the partition function in a reasonable amount of computer time, but it can also just as easily be used to estima te the probability of held-out documents for directed topic models, including Latent Dirichlet Allo cation (for”
Rethinking LDA: Why Priors Matter (2009, unknown): “[17].”
Sparse Additive Generative Models of Text (2011, methodology): “1 Discriminative methods may yield better performance on the document classiﬁcation task, but our goal here is 1 Wallach et al (2009a) ﬁnd that asymmetric Dirichlet priors for term distributions provide no advantage over symmetric priors. to compare generative models which are amenable to…”
Reading Tea Leaves: How Humans Interpret Topic Models (2009, background): “[13] provide a summary of evaluation techniques.”
Structured Stochastic Variational Inference (2014, methodology): “We estimated marginal log probabilities with a Chib-style estimator (Wallach et al., 2009).”
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling (2018, methodology): “In order to capture the hierarchical document latent representation, WHAI uses the Poisson gamma belief network (PGBN) of Zhou et al. (2016), a deep probabilistic topic model, as the generative network (encoder).”
Automatic Evaluation of Topic Coherence (2010, methodology): “ety of ways, and evaluated intrinsically in terms of model perplexity (Wallach et al., 2009), but there has been less effort on qualitative understanding of the semantic nature of the learned topics.”
ZhuSuan: A Library for Bayesian Deep Learning (2017, methodology): “Even if not used for learning, importance sampling is still a simple and eﬃcient method for evaluating marginal log likelihoods of latent variable models (Wallach et al., 2009).”
Topic modelling for qualitative studies (2017, methodology): “…at perplexity [39] on the original dataset or measure the predictive power of a model by computing the log probability of a held-out set of documents that are shown to work not very well [40]; an important recent study advocates a Bayesian approach based on posterior predictive checking over [41].”
Topic Modeling Using Latent Dirichlet allocation (2021, background): “The aforementioned research effort advocates that the quality of a topic is often determined by the coherence [1, 101, 145] of its constituent words and its relative importance [1, 69] to the analysis task in comparison to other topics [70].”
Topic Modeling in Embedding Spaces (2019, methodology): “The ETM provides good predictive performance, as measured by held-out log-likelihood on a document completion task (Wallach et al., 2009).”