Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression

By David Mimno, Andrew McCallum. Conference on Uncertainty in Artificial Intelligence, 2008

432 citations recorded by the source corpus.

Although fully generative models have been successfully used to model the contents of text documents, they are often awkward to apply to combinations of text data and document metadata. In this paper we propose a Dirichlet-multinomial regression (DMR) topic model that includes a log-linear prior on document-topic distributions that is a function of observed features of the document, such as author, publication venue, references, and dates. We show that by selecting appropriate features, DMR topic models can meet or exceed the performance of several previously published topic models designed for specific data.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f53e0ec3563c5e32821c65c49c1c41208861751e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Why people hate your app: making sense of user feedback in a mobile app store (2013, background): “Topic models [6, 5, 17, 18] have been widely used to find meaningful topics (i.”
Demographic Dialectal Variation in Social Media: A Case Study of African-American English (2016, methodology): “This model is also related to topic models where the selection of θ conditions on covariates (Mimno and McCallum, 2008; Ramage et al., 2011; Roberts et al., 2013), though it is much simpler without full latent topic learning.”
An overview of topic modeling and its current applications in bioinformatics (2016, background): “A Dirichlet-multinomial regression (DMR) topic model (Mimno and McCallum 2012) provides a log-linear prior for document-topic distributions, and its aim is to incorporate arbitrary types of observed document features, such as author and publication venue.”
Learning Latent Personas of Film Characters (2013, methodology): “To incorporate observed metadata in the form of movie genre, character age and character gender, we adopt an “upstream” modeling approach (Mimno and McCallum, 2008), letting those observed features influence the conditional probability with which a given character is expected to assume a particular ”
DiscLDA: Discriminative Learning for Dimensionality Reduction and Classification (2008, background): “Mimno and McCallum [8] proposed a Dirichlet-multinomial regression which can handle variou s types of side information, including the case in which this side information is an indicator variable of the class ( y).”
Replicated Softmax: an Undirected Topic Model (2009, background): “Recently, [11] introdu ced a Dirichlet-multinomial regression model, where a prior on the document-specific topic distribu ions was modeled as a function of observed metadata of the document.”
Multiple domain user personalization (2011, methodology): “λ however is difficult to sample, and [11] optimizes it using L-BFGS.”
MedLDA: maximum margin supervised topic models (2012, methodology): “The model presented by (Mimno and McCallum, 2008) is also an upstream model for incorporating document meta-features.”
Dynamic Language Models for Streaming Text (2014, background): “Mimno and McCallum (2008) and Eisenstein et al. (2010) similarly conditioned text on observable features (e.g., author, publication venue, geography, and other document-level metadata), but conducted inference in a batch setting, thus their approaches are not suitable for streaming data.”
A Semantic Approach for Estimating Consumer Content Preferences from Online Search Queries (2018, methodology): “…same web page, we use the average topic intensities across these queries, which is denoted as θq(p) ∑qθqlpq/∑qlpq in the following equation.2 Following the Dirichlet-multinomial regression topic model (Mimno and McCallum 2008), we assume that θp ~DirichletK(exp(RT1θq(p)), . . . , exp(RTKθq(p))).”
Modeling Documents with Deep Boltzmann Machines (2013, methodology): “Directed graphical models, such as Latent Dirichlet Allocation (LDA), CTM, H-LDA, have been extensively used for this [3, 2, 8].”
Mining Text Data (2012, methodology): “Therefore, the method discussed in [48] uses a new representation, which is referred to as the bursty feature representation for mining text streams.”
Discovering Urban Functional Zones Using Latent Activity Trajectories (2015, methodology): “As a result, TF-IDF-based methods forms seven clusters (c 0 –c 6 ) while LDA-based methods and DMR-based methods form nine functional zones.”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, methodology): “Incorporating Metadata Document metadata has been commonly used to improve the quality of topic models (Mimno and McCallum, 2012; Ramage et al., 2009; Zhu et al., 2012), and previous works have used metadata for adapting RNN-based language models (Jaech and Ostendorf, 2018) or learning better docume”
Applications of Topic Models (2017, background): “, 2007a], a prior learned from observed sentiment [Mimno and McCallum, 2008], or from a latent variable that can serve as a proxy for sentiment [Lin and He, 2009].”
stm: An R Package for Structural Topic Models (2019, background): “…gratefully acknowledge grant support from the Spencer Foundation’s New Civics initiative, the Hewlett Foundation, a National Science extended these (Mimno and McCallum 2008; Socher et al. 2009; Eisenstein et al. 2010; Rosen-Zvi et al. 2010; Quinn et al. 2010; Ahmed and Xing 2010; Grimmer 2010;…”