A correlated topic model of Science
By David M. Blei, John D. Lafferty. 2007
1,579 citations recorded by the source corpus.
Topic models, such as latent Dirichlet allocation (LDA), can be useful tools for the statistical analysis of document collections and other discrete data. The LDA model assumes that the words of each document arise from a mixture of topics, each of which is a distribution over the vocabulary. A limitation of LDA is the inability to model topic correlation even though, for example, a document about genetics is more likely to also be about disease than X-ray astronomy. This limitation stems from the use of the Dirichlet distribution to model the variability among the topic proportions. In this paper we develop the correlated topic model (CTM), where the topic proportions exhibit correlation via the logistic normal distribution [J. Roy. Statist. Soc. Ser. B 44 (1982) 139--177]. We derive a fast variational inference algorithm for approximate posterior inference in this model, which is complicated by the fact that the logistic normal is not conjugate to the multinomial. We apply the CTM to the articles from Science published from 1990--1999, a data set that comprises 57M words. The CTM gives a better fit of the data than LDA, and we demonstrate its use as an exploratory tool of large document collections.
Subfields: Artificial Intelligence, General Social Sciences
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Jianfeng Gao
- Mohit Bansal
- Ruslan Salakhutdinov
- Percy Liang
- Sergey Levine
- Quoc V. Le
- Kyunghyun Cho
In General Social Sciences
- David M. Blei
- Chengxiang Zhai
- Charu C. Aggarwal
- Brandon M Stewart
- David Mimno
- Andrew McCallum
- Margaret E. Roberts
- Chong Wang
Citations in the corpus
- Variational Inference: A Review for Statisticians (2016, background): “Blei and Lafferty (2007) later adapted their idea to nonconjugate topic models, and researchers have continued to improve the original bound (Khan et al. 2010; Marlin, Khan, and Murphy 2011; Ermis and Bouchard 2014).”
- Deep Exponential Families (2014, background): “One hundred topics fall into the range of topics searched in the topic modeling literature [4].”
- Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions (2015, neutral): “…0 < γ < 2 is a scalar) and fixed batch-size M k = M for all k with a stochastic gradient ∇ f ( λ ) , then we have where c is a constant such that c > 1 / (2 α ) and α ∗ := α − 1 / (2 c ) . noise The expectation is taken with respect to the ξ := { ξ 0 , ξ 1 , . . . , ξ t − 1 } , and a random…”
- Rethinking LDA: Why Priors Matter (2009, background): “There is a significant body of work applying LDA to an wide variety of tasks including analysis of news articles [14], study of the history of scientific ideas [2, 9], topic-based search interfaces1 and navigation tools for digital libraries [12].”
- Stochastic variational inference (2012, background): “For example, nonconjugate models can capture correlations between topics (Blei and Lafferty, 2007) or topics changing over time (Blei and Lafferty, 2006; Wang et al., 2008), and the general algorithm presented here cannot be used in these settings.”
- Learning Topic Models -- Going beyond SVD (2012, background): “See for example the Correlated Topic Model (CTM) [8] and the Pachinko Allocation Model (PAM) [26].”
- Deep Poisson Factor Modeling (2015, neutral): “Several models have been proposed recently to address the limitations described above [1, 2, 6, 27].”
- Variational inference in nonconjugate models (2012, positive): “We compared it to the original inference algorithm of Blei and Lafferty (2007).”
- A Practical Algorithm for Topic Modeling with Provable Guarantees (2012, background): “Examples of such distributions include latent Dirichlet allocation [Blei et al., 2003], correlated topic models [Blei & Lafferty, 2007], and Pachinko allocation [Li & McCallum, 2007].”
- A General Method for Robust Bayesian Modeling (2015, methodology): “We follow the metric used in recent topic modeling literature (Blei and Lafferty, 2007; Asuncion et al., 2009; Wang et al., 2011; Hoffman et al., 2013), where we hold out part of a document and predict its remainder.”
- Topic regression multi-modal Latent Dirichlet Allocation for image annotation (2010, methodology): “To this end, we draw insights from several recent topic models [3] [11] and adopt a linear Gaussian regression module which takes in the image topic proportion as its input and target the hidden topic proportion for annotation texts as the response variable (in line with the task of predicting…”
- Non-conjugate Variational Message Passing for Multinomial and Binary Regression (2011, methodology): “The messages are still not conjugate, so some numerical method must still be used to learn m and v: while [3] used LBFGS we will use NCVMP.”