Structural Topic Models for Open‐Ended Survey Responses
By Margaret E. Roberts, Brandon M Stewart, Dustin Tingley, Christopher Lucas, Jetson Leder-Luis, S. Gadarian, B. Albertson, David G. Rand. 2014
1,756 citations recorded by the source corpus.
Collection and especially analysis of open-ended survey responses are relatively rare in the discipline and when conducted are almost exclusively done through human coding. We present an alternative, semiautomated approach, the structural topic model (STM) (Roberts, Stewart, and Airoldi 2013; Roberts et al. 2013), that draws on recent developments in machine learning based analysis of textual data. A crucial contribution of the method is that it incorporates information about the document, such as the author's gender, political affiliation, and treatment assignment (if an experimental study). This article focuses on how the STM is helpful for survey researchers and experimentalists. The STM makes analyzing open-ended responses easier, more revealing, and capable of being used to estimate treatment effects. We illustrate these innovations with analysis of text from surveys and experiments.
Subfields: General Social Sciences, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In General Social Sciences
- Brandon M Stewart
- Margaret E. Roberts
- Dustin Tingley
- Zachary C. Steinert-Threlkeld
- Matthew J. Denny
- Arthur Spirling
- Katherine A. Keith
- Brendan T. O'Connor
In Information Extraction
- Zachary C. Steinert-Threlkeld
- Roi Reichart
- Brendan T. O'Connor
- Dan Jurafsky
- Margaret E. Roberts
- Brandon M Stewart
Citations in the corpus
- CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “This is usually done when the treatment affects the text, and the model architecture is manipulated to incorporate the treatment assignment (Roberts et al. 2014; Johansson, Shalit, and Sontag 2016; Roberts, Stewart, and Nielsen 2018).”
- Handling Bias in Toxic Speech Detection: A Survey (2022, background): “(STM [70]) on the Founta100k dataset and identified a latent topic containing terms prevalent in AAE (e.”
- Quantitative analysis of large amounts of journalistic texts using topic modelling (2016, background): “Dynamic Topic Models (Blei and Lafferty 2006) and Structural Topic Models (Roberts et al. 2014) are two extensions of LDA that explicitly deal with shifts over time and between groups of documents, respectively.”
- The Diversity–Innovation Paradox in Science (2019, methodology): “For this, we use natural language processing techniques of phrase extraction and structural topic modeling (22, 23) to identify terms representing substantive concepts in millions of documents (# concepts, mean = 56.”
- Twitter as Data (2018, methodology): “If conducting text analysis, any of these approaches enable a structural topic model (Roberts et al., 2014).”
- Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It (2017, background): “2010), how Senators vary in their priorities and credit claiming (Grimmer 2010), how Japanese politicians increasingly focus on foreign policy (Catalinac 2016) or how citizens differ in terms of the themes they emphasize in open-ended survey questions (Roberts et al. 2014).”
- stm: An R Package for Structural Topic Models (2019, methodology): “In the appendix of Roberts et al. (2014) we address this concern using both a simulation and a permutation test approach.”
- Beliefs about Climate Beliefs: The Importance of Second-Order Opinions for Climate Politics (2017, background): “77see Roberts et al. 2014 for additional details The bottom half of the figure, which is our core interest, plots the relationship between topics and respondents’ own views on whether climate change is happening.”
- Prescriptive analytics: Literature review and research challenges (2020, neutral): “…making problems in different contexts are: Artificial Neural Networks (Domingos, Burguillo, & Lenaerts, 2017), Bayesian Networks (Said, Shahzad, Zamai, Hubac, & Tollenaere, 2016), topic models (Roberts et al., 2014), fuzzy sets (Pal & Ceglarek, 2013), Markov models (Bousdekis et al., 2018).”
- How to make causal inferences using texts (2018, positive): “Examples in prior work tends to explicitly take the view of g(Y) as a latent variable about which there is some additional measurement uncertainty and advocated approaches to incorporate this additional uncertainty into our confidence intervals (Roberts et al., 2014; Fong and Grimmer, 2016).”
- Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates (2020, background): “(2) Topic models are generative probabilistic models that learn latent topics in document collections and represent documents as distributions over topics (Blei et al., 2003; Boyd-Graber et al., 2014; Roberts et al., 2014).”