Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality

By Jey Han Lau, David Newman, Timothy Baldwin. 2014

650 citations recorded by the source corpus.

Topic models based on latent Dirichlet allocation and related methods are used in a range of user-focused tasks including document navigation and trend analysis, but evaluation of the intrinsic quality of the topic model and topics remains an open research area. In this work, we explore the two tasks of automatic evaluation of single topics and automatic evaluation of whole topic models, and provide recommendations on the best strategy for performing the two tasks, in addition to providing an open-source toolkit for topic and topic model evaluation.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Graph-Sparse LDA: A Topic Model with Structured Sparsity (2014, background): “[20] introduced the idea of “intrusion detection” where they hypothesized that a mor e coherent, or interpretable, topic would be one where a human annotator would be able to identify an inserted “intr ude ” word among the top 5 words in a topic; [21] automated this process.”
Topic Modeling in Embedding Spaces (2019, background): “It is the average pointwise mutual information of two words drawn randomly from the same document (Lau et al., 2014), where { w 10 } denotes the top-10 most likely words in topic k .”
Discovering Discrete Latent Topics with Neural Variational Inference (2017, background): “We observe that in both the GSB and RSB, the NPMI scores of the former topics in the stick breaking order are higher than the latter ones.”
Topic Compositional Neural Language Model (2017, methodology): “Following Lau et al. (2014), we compute topic coherence using normalized PMI (NPMI).”
Topic Modeling Using Latent Dirichlet allocation (2021, background): “The aforementioned research effort advocates that the quality of a topic is often determined by the coherence [1, 101, 145] of its constituent words and its relative importance [1, 69] to the analysis task in comparison to other topics [70].”
Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge (2016, background): “Since the HA/DR lexicon labels are the result of expert knowledge and crowd-sourcing, the purity provides us with a measure of semantic topic consistency similar to word intrusion tests (Chang et al., 2009; Lau et al., 2014).”
Interpretable Machine Learning : Lessons from Topic Modeling (2016, background): “Metrics to measure the coherence of topics were introduced by [17] and [16], and later improved upon and generalized by others [13] [21].”
Predicting In-Game Actions from Interviews of NBA Players (2019, methodology): “We train a topic model 20 and optimize the number of topics on our entire training set I train for maximal coherence (Lau, Newman, and Baldwin 2014), resulting in a set Z of 36 topics.”
Topically Driven Neural Language Model (2017, methodology): “Following Lau et al. (2014), we compute topic coherence using normalised PMI (“NPMI”) scores.”
Autoencoding Variational Inference For Topic Models (2017, methodology): “Lau et al. (2014) showed that among all the competing metrics, normalized pointwise mutual information (NPMI) between all the pairs of words in a set of topics matches human judgment most closely, so we adopt it in this work.”
A Model of Text for Experimentation in the Social Sciences (2016, methodology): “Semi-automated approaches using a series of crowd-sourced experiments on Amazon’s Mechanical Turk are a promising strategy to fully automate human evaluation of models of text (Lau et al. 2014).”
An analysis of the coherence of descriptors in topic modeling (2015, methodology): “…to correlate human judgements with our automated results, although this would likely be different to prior coherence studies that requested ratings of individual topics, (Newman et al., 2010; Mimno et al., 2011; Lau et al., 2014), where descriptor comparisons would instead need to be considered.”
Dirichlet Variational Autoencoder (2019, methodology): “Two popular performance measures in the topic model ﬁelds, which are perplexity and topic coherence via normalized pointwise mutual information (NPMI) (Lau et al., 2014), have been used with 20Newsgroups and RCV1-v2 datasets.”
Exploring the Space of Topic Coherence Measures (2015, methodology): “Subsequent empirical works on topic coherence [12, 18, 10] proposed a number of measures based on word statistics that differ in several details: definition, normalization and aggregation of word statistics and reference corpus.”
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence (2020, background): “Coherence can be measured in numerous ways, from human evaluation via intrusion tests (Chang et al., 2009) to approximated scores (Lau et al., 2014; R¨oder et al., 2015).”
BERTopic: Neural topic modeling with a class-based TF-IDF procedure (2022, background): “This coherence measure has been shown to emulate human judgment with reasonable performance (Lau et al., 2014).”