Worth Weighting? How to Think About and Use Weights in Survey Experiments

By Luke W. Miratrix, Jasjeet S. Sekhon, Alexander G. Theodoridis, Luis F. Campos. Political Analysis, 2017

169 citations recorded by the source corpus.

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Subfields: Sociology and Political Science, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/34cfdf6de4564d923e57acf0bf6d6808a803372e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Sociology and Political Science

In Statistics and Probability

Citations in the corpus

Improving the External Validity of Conjoint Analysis: The Essential Role of Profile Distribution (2021, unknown): “We note that our focus is on the external validity of conjoint profiles , and is distinct from another important issue of representativeness of respondents in survey experiments (see, e.g., Mutz, 2011; Mullinix et al. , 2015; Coppock, Leeper, and Mullinix, 2018; Miratrix et al. , 2018).”
Reducing Exclusionary Attitudes through Interpersonal Conversation: Evidence from Three Field Experiments (2020, background): “As [11] note, “The PATE can only be different from the SATE when two things hold: (1) there is meaningful variation in the treatment impact, and (2) that variation is correlated with the weights.”
Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents (2019, background): “If these important moderators are measured in the sample and are known in the population, then SATEs can be reweighted to estimate PATEs (Franco et al., 2017, Miratrix et al., 2018).”