Misunderstandings between experimentalists and observationalists about causal inference

By Kosuke Imai, Gary King, Elizabeth A. Stuart. 2008

855 citations recorded by the source corpus.

Summary. We attempt to clarify, and suggest how to avoid, several serious misunderstandings about and fallacies of causal inference. These issues concern some of the most fundamental advantages and disadvantages of each basic research design. Problems include improper use of hypothesis tests for covariate balance between the treated and control groups, and the consequences of using randomization, blocking before randomization and matching after assignment of treatment to achieve covariate balance. Applied researchers in a wide range of scientific disciplines seem to fall prey to one or more of these fallacies and as a result make suboptimal design or analysis choices. To clarify these points, we derive a new four‐part decomposition of the key estimation errors in making causal inferences. We then show how this decomposition can help scholars from different experimental and observational research traditions to understand better each other's inferential problems and attempted solutions.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/edb9982c997c88f37cf0e8b55e1ef265586874d4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Propensity score matching and variations on the balancing test (2011, neutral): “Despite the widespread practice of using hypothesis tests to evaluate balance in the econometric literature (e.g., see Smith and Todd 2005a); Imai et al. (2008) have an alternate view and describe this as a “balancing test fallacy” that should be avoided.”
Regression-adjusted average treatment effect estimates in stratified randomized experiments (2019, neutral): “Recently, stratified randomized experiments were discussed under the Neyman–Rubin model and randomizationbased inference framework [14, 16, 18, 28, 32], however no formal asymptotic results have been established.”
An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies (2011, methodology): “…authors have frequently used statistical significance testing to compare the mean of continuous covariates or the distribution of categorical variables between treated and untreated subjects in propensity score matched samples authors for two reasons (Imai et al., 2008; Austin, 2008b, 2009b).”
Does Marriage Reduce Crime? (2015, background): “While balance ideally is assessed by comparing the joint empirical distribution of all covariates X between the matched and control groups, this becomes infeasible with high-dimensional X (Imai, King, and Stuart 2008, p. 495).”
Evaluating continuous training programmes by using the generalized propensity score (2012, background): “we might observe for some covariates decreased tstatistics which might be driven by increased variances and not by decreased mean differences (see Imai, King and Stuart (2008) for a general discussion of balance test fallacy in the context of matching approaches).”
Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments (2008, methodology): “We would benefit from further development of ways to create and assess balance (e.g., Imai, King, and Stuart 2008; Sekhon 2007), as well as from better-justified standards for how much balance should be achieved.”
In Pursuit of Balance: Randomization in Practice in Development Field Experiments (2008, positive): “Statistical efficiency is greatest when the variables chosen are strongly related to the outcome of interest (Imai et al., 2008).”
Statistical Analysis of List Experiments (2012, methodology): “(Imai, King, and Stuart 2008).”
The role of prediction modeling in propensity score estimation: an evaluation of logistic regression, bCART, and the covariate-balancing propensity score. (2014, background): “It is common practice to fit a PS model, check covariate balance, and then reestimate the PS model using different functional forms and interactions until an acceptable degree of balance is achieved (11).”
Do Democracies Make Inferior Counterinsurgents? Reassessing Democracy's Impact on War Outcomes and Duration (2010, background): “…types of incumbent force structures while acknowledging that variables have different effects across these eras+ See Lyall and Wilson III 2009+ 69+ See Imai, King, and Stuart 2008; Sekhon 2006; and Morgan and Winship 2007, 116–21+ 70+ Ho et al+ 2007, 23, n+15+ Standardized bias is the…”
Prognostic score–based balance measures for propensity score methods in comparative effectiveness research (2013, neutral): “Other common balance measures include Kolmogorov-Smirnov test statistics and t-statistics, although caution is urged when using measures that conflate sample size and balance, such as hypothesis tests [16].”
Balance diagnostics after propensity score matching. (2019, background): “In other words, the statistical insignificance can occur when we randomly drop more number of controls (11).”
Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples (2009, methodology): “Comparing the distributions of the estimated propensity score between treated and untreated subjects Some authors have proposed that the distribution of the estimated propensity score be compared between treated and untreated subjects in the propensity-score matched sample [12, 29].”