Incorporating Domain Knowledge into Topic Modeling via Dirichlet Forest Priors

By David Andrzejewski, Xiaojin Zhu, Mark W. Craven. International Conference on Machine Learning, 2009

479 citations recorded by the source corpus.

Users of topic modeling methods often have knowledge about the composition of words that should have high or low probability in various topics. We incorporate such domain knowledge using a novel Dirichlet Forest prior in a Latent Dirichlet Allocation framework. The prior is a mixture of Dirichlet tree distributions with special structures. We present its construction, and inference via collapsed Gibbs sampling. Experiments on synthetic and real datasets demonstrate our model's ability to follow and generalize beyond user-specified domain knowledge.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/268b37db3acb544de18348575b051a18e32033ab

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erik Cambria
2. Soujanya Poria
3. Bing Liu
4. B. Liu
5. Percy Liang
6. Mohit Bansal
7. Noah A. Smith
8. Shuai Wang

Citations in the corpus

• Sentence-Based Text Analysis for Customer Reviews (2016, background): “• Incorporating informative word-topic probabilities consistent with domain knowl- edge through the prior distribution (Andrzejewski et al., 2009).”
• Mining topics in documents: standing on the shoulders of big data (2014, methodology): “The existing knowledgebased topic model DF-LDA [1] cannot distinguish multiple senses because its definition of must-link is transitive.”
• Topic Modeling using Topics from Many Domains, Lifelong Learning and Big Data (2014, methodology): “DF-LDA (Andrzejewski et al., 2009): A knowledge-based topic model that can use the user-provided knowledge.”
• Evaluating Topic Coherence Using Distributional Semantics (2013, methodology): “2 Related work Andrzejewski et al. (2009) proposed a method for generating coherent topics which used a mixture of Dirichlet distributions to incorporate domain knowledge.”
• Mining Aspect-Specific Opinion using a Holistic Lifelong Topic Model (2016, methodology): “Since our LAST model can exploit prior aspect and opinion knowledge, it is thus related to knowledge-based topic models such as [2, 18, 41, 33, 17].”
• A Survey of Human-in-the-loop for Machine Learning (2021, background): “[91] extend a statistical framework [100] to allow end-users to add, remove, or change the weights of words within each topic.”
• Artificial intelligence in marketing: Topic modeling, scientometric analysis, and research agenda (2020, methodology): “Because the vast amount of information available goes beyond the processing capacities of humans (Andrzejewski et al., 2009; Wallach, 2006; Wang and Blei, 2011), topic models are valuable.”
• Interactive Topic Modeling (2011, methodology): “…doesn’t make any sense at all (Newman et al., 2010); this topic shouldn’t be associated with this document but is (Ramage et al., 2009); these words shouldn’t be the in same topic but are (Andrzejewski et al., 2009); or these words should be in the same topic but aren’t (Andrzejewski et al., 2009).”
• Graph-Sparse LDA: A Topic Model with Structured Sparsity (2014, methodology): “[32] use Dirichlet forest priors to enforce expert-provided “must be in same topic” and “cannot be in same topic” constraints between words.”
• Interpretable Machine Learning : Lessons from Topic Modeling (2016, background): “Work has been done to design topic models that can incorporate human preferences, such as specifying which words should or should not appear in the same topic [2], or using seed words to guide topics [12] [18].”
• Topic modelling for qualitative studies (2017, methodology): “First, in the Topic-in-Set knowledge model and its extension with Dirichlet forest priors [30, 31], words are assigned with ‘z-labels’.”
• Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge (2016, methodology): “The second model from Andrezejewski and Zhu allows specification of Must-Link and Cannot-Link relationships between words that help partition otherwise muddled topics (Andrzejewski et al., 2009).”
• Incorporating Lexical Priors into Topic Models (2012, background): “The Must and Cannot links required for the supervision (Andrzejewski et al., 2009) are automatically obtained by adding a must-link between every pair of words belonging to the same seed set and a split constraint between every pair of words belonging to different sets.”
• Optimizing Semantic Coherence in Topic Models (2011, unknown): “Although there has been some work in semi-supervised contexts on constructing models that avoid semantic coherence problems [1], we are not aware of any work that relies only the unstructured documents being modeled.”