Automatic Evaluation of Topic Coherence

By David Newman, Jey Han Lau, Karl Grieser, Timothy Baldwin. North American Chapter of the Association for Computational Linguistics, 2010

1,113 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/8e31f3c7e70e9a5f8afafd86cebc004d5eca8c2b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Improving LDA topic models for microblogs via tweet pooling and automatic labeling (2013, methodology): “It is possible to automatically measure topic coherence with near-human accuracy [10] using a score based on pointwise mutual information (PMI).”
Statistical Model Aggregation via Parameter Matching (2019, unknown): “[7] in terms of the UCI coherence score [27], −2.”
Short Text Topic Modeling Techniques, Applications, and Performance: A Survey (2019, background): “A lot of metrics have been proposed for measuring the coherence of topics in texts [59], [60].”
Autoencoding Variational Inference For Topic Models (2017, methodology): “Traditionally, perplexity has been used to measure the goodness-of-fit of the model but it has been repeatedly shown that perplexity is not a good metric for qualitative evaluation of topics (Newman et al., 2010).”
Graph-Sparse LDA: A Topic Model with Structured Sparsity (2014, background): “[22] and [23] developed me asures of topic coherence that strongly correlated with human annotations of topic quality.”
Exploring the Space of Topic Coherence Measures (2015, methodology): “The coherences of [12, 13, 1] can be written as:”
Topic Modeling using Topics from Many Domains, Lifelong Learning and Big Data (2014, methodology): “It has also been used to evaluate topic models (Newman et al., 2010).”
Interactive Machine Learning via a GPU-accelerated Toolkit (2017, background): “On the other hand, automatic evaluation of topic quality [29] is complicated and users may have different criteria at the same time.”
Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality (2014, methodology): “Newman et al. (2010) capture topic inter-pretability using a more direct approach, by asking human users to rate topics (represented by their top-10 topic words) on a 3-point scale based on how coherent the topic words are (i.e. their observed coherence).”
Interactive Topic Modeling (2011, background): “…III, 2009); this topic should have syntactic coherence (Gruber et al., 2007; Boyd-Graber and Blei, 2008); this topic doesn’t make any sense at all (Newman et al., 2010); this topic shouldn’t be associated with this document but is (Ramage et al., 2009); these words shouldn’t be the in same topic…”
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation (2017, background): “As a result, researchers have proposed topic coherence measures, which are a qualitative approach to automatically uncover the coherence of a topic [18], [33], and the underlying idea is rooted in the distributional hypothesis of linguistics [22]; words with similar meanings tend to occur in similar”
Interpretable Machine Learning : Lessons from Topic Modeling (2016, background): “Metrics to measure the coherence of topics were introduced by [17] and [16], and later improved upon and generalized by others [13] [21].”
Evaluation methods for unsupervised word embeddings (2015, background): “Manual evaluation is expensive and time-consuming, but other work establishes that automated evaluations can closely model human intuitions (Newman et al., 2010).”
Topically Driven Neural Language Model (2017, methodology): “…a powerful tool for extracting the macro-level content structure of a document collection in the form of the latent topics (usually in the form of multinomial distributions over terms), with a plethora of applications in NLP (Hall et al., 2008; Newman et al., 2010a; Wang and McCallum, 2006).”
stm: An R Package for Structural Topic Models (2019, background): “Semantic coherence is a criterion developed by Mimno et al. (2011) and is closely related to pointwise mutual information (Newman et al. 2010): it is maximized when the most probable words in a given topic frequently co-occur together.”
BTM: Topic Modeling over Short Texts (2014, background): “To evaluate the quality of topics discovered, we calculated the average PMI-Score, i.e., 1K P k PMI-ScoreðkÞ, for each method.”