Reading Tea Leaves: How Humans Interpret Topic Models

By Jonathan D. Chang, Jordan L. Boyd-Graber, Sean Gerrish, Chong Wang, David M. Blei. 2009

2,538 citations recorded by the source corpus.

Subfields: Information Extraction, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Extraction

In Human-Computer Interaction

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “The methodology follows earlier work on evaluating the interpretability of probabilistic topic models with intrusion tasks (Chang et al., 2009).”
Towards A Rigorous Science of Interpretable Machine Learning (2017, neutral): “The common intrusion-detection test [Chang et al., 2009] in topic models is a form of the forward simulation/prediction task: we ask the human to ﬁnd the diﬀerence between the model’s true output and some corrupted output as a way to determine whether the human has correctly understood what the…”
The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification (2014, methodology): “In our view, there are at least three widely known types of interpretable models: sparse linear classifiers ([30, 8, 31]); discretization methods, such as decision trees and decision lists (e.”
Sparse Overcomplete Word Vector Representations (2015, methodology): “We also evaluate our word vectors in a word intrusion experiment with humans (Chang et al., 2009) and ﬁnd that our sparse vectors are more interpretable than the original vectors ( § 4).”
Graph-Sparse LDA: A Topic Model with Structured Sparsity (2014, background): “(Chang et al. 2009) measured interpretability by how easily a human could identify an inserted “intruder” word among the top 5 words in a topic.”
Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2020, background): “This aspiration to evaluate inferred associations with users is similar to that of evaluating latent topics for topic models (Chang et al., 2009).”
Full-Text or Abstract? Examining Topic Coherence Scores Using Latent Dirichlet Allocation (2017, background): “However, such a measure correlates negatively with human interpretability [32], making topics with high predictive likelihood less coherent from a human perspective.”
Topically Driven Neural Language Model (2017, background): “There are various ways to estimate test perplexity (Wallach et al., 2009), but Chang et al. (2009) show that perplexity does not correlate with the coherence of the generated topics.”
Topic Modeling in Management Research: Rendering New Theory from Textual Data (2019, background): “However, Chang et al. (2009) pointed to disparities between some quantitative metrics and how people interpret topics: topic models that perform better on quantitative metrics tend to infer topics that humans judge to be semantically less meaningful.”
Soft Ngram Representation and Modeling for Protein Remote Homology Detection (2017, background): “For example, the class of approaches known as topic models, originally introduced for natural language processing, have been widely employed in several scientific fields [17], [18], [19], motivated by their effectiveness and expressiveness in dealing with large datasets [20].”
Mining Text Data (2012, methodology): “Corpus Summarization: Clustering techniques provide a coherent summary of the collection in the form of cluster-digests [83] or word-clusters [17, 18], which can be used in order to provide summary insights into the overall content of the underlying corpus.”
A Neural Autoregressive Topic Model (2012, methodology): “It’s interesting to note that the detection of such intruder words has been used previously as a task in user studies to evaluate the quality of the topics learned by LDA, though at the level of single topics and not whole documents [8].”
Rotated Word Vector Representations and their Interpretability (2017, background): “(Murphy et al., 2012; Sun et al., 2016) In the standard word intrusion task, human eval-uators pick out the intruder words, and the results report the accuracy of the evaluators (Chang et al., 2009).”
Automatically Inferring Gender Associations from Language (2019, methodology): “To test our clusters, we employed the Word Intrusion task (Chang et al., 2009).”
Computer-Assisted Text Analysis for Comparative Politics (2015, background): “Furthermore, human evaluation experiments in the computer science literature (Chang et al., 2009) has shown that models which provide the best statistical fit can be less interpretable.”
Short Text Topic Modeling Techniques, Applications, and Performance: A Survey (2019, background): “However, this likelihood is not necessarily a good indicator of the quality of the extracted topics [61].”
Topic modelling for qualitative studies (2017, methodology): “Therefore, in this experiment we asked the subjects direct questions about understanding topics rather than used subjects as oracles to judge objective topic qualities, as has been done in the experiments conducted in Chang et al. [38] and as we will do in Section 3.4.”