Demystifying Double Robustness: A Comparison of Alternative Strategies for Estimating a Population Mean from Incomplete Data

By Joseph Kang, Joseph L. Schafer. 2007

1,503 citations recorded by the source corpus.

When outcomes are missing for reasons beyond an investigator's control, there are two different ways to adjust a parameter estimate for covariates that may be related both to the outcome and to missingness. One approach is to model the relationships between the covariates and the outcome and use those relationships to predict the missing values. Another is to model the probabilities of missingness given the covariates and incorporate them into a weighted or stratified estimate. Doubly robust (DR) procedures apply both types of model simultaneously and produce a consistent estimate of the parameter if either of the two models has been correctly specified. In this article, we show that DR estimates can be constructed in many ways. We compare the performance of various DR and non-DR estimates of a population mean in a simulated example where both models are incorrect but neither is grossly misspecified. Methods that use inverse-probabilities as weights, whether they are DR or not, are sensitive to misspecification of the propensity model when some estimated propensities are small. Many DR methods perform better than simple inverse-probability weighting. None of the DR methods we tried, however, improved upon the performance of simple regression-based prediction of the missing values. This study does not represent every missing-data problem that will arise in practice. But it does demonstrate that, in at least some settings, two wrong models are not better than one.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Detecting and Correcting for Label Shift with Black Box Predictors (2018, background): “Even in the simpler problem of average treatment effect (ATE) estimation, it’s known that using estimated propensity can lead to estimators with large variance [12].”
Balanced Policy Evaluation and Learning (2017, background): “(1) By taking a simple plug-in approach for a nuisance parameter (propensities) that appears in the denominator, existing weighting-based methods are either subject to very high variance or must rely on scarcely-motivated stopgap measures such as clipping (see also [27]).”
Approximate residual balancing: debiased inference of average treatment effects in high dimensions (2016, methodology): “…0:95; AIPW , which pairs elastic net regression adjustments with the above inverse propensity weights (9); the weighted elastic net, motivated by Kang and Schafer (2007), that uses inverse propensity weights as sample weights for the elastic net regression; targeted maximum likelihood…”
Doubly Robust Policy Evaluation and Optimization (2014, methodology): “Doubly robust estimation is widely used in statistical inference (see, e.g., Kang and Schafer, 2007, and the references therein).”
Doubly Robust Policy Evaluation and Learning (2011, methodology): “Furthermore, most of previous analysis of doubly robust estimation studies asymptotic behavior or relies on various modeling assumptions (e.g., Robins et al. (1994), Lunceford & Davidian (2004), and Kang & Schafer (2007)).”
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning (2019, background): “[12] also experimentally showed that the performance of β̂dr might be very bad in practice when the model is wrong.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “These ideas have also been employed in reinforcement learning (Kang et al., 2007; Bang and Robins, 2005; Dudı́k et al., 2011).”
Learning Representations for Counterfactual Inference (2016, background): “Once the treatment assignment probability has to be estimated, as is the case in most observational studies, their efﬁcacy might wane considerably (Kang & Schafer, 2007).”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, neutral): “Similar phenomenon was observed in the statistics literature [29].”
Simple Rules for Complex Decisions (2017, methodology): “is ignorability assumption is unavoidable, and is similarly required for methods based on propensity scores [1, 7, 16, 27, 28, 30, 31].”
NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE (2016, background): “Furthermore, simulation studies have shown that IPW is heavily reliant on correct specification of the propensity score model, and that bias and imprecision are increased by misspecification.(25)”
Sensitivity analysis for inverse probability weighting estimators via the percentile bootstrap (2017, background): “This is most likely to be due to the wellknown phenomenon that the IPW estimators are unstable when the selection probability is close to 0 (Kang and Schafer, 2007) and the relatively small sample size being used (n = 200).”
Bayesian Nonparametric Modeling for Causal Inference (2011, background): “Estimators with this property are called “doubly robust” or “doubly protected” and are discussed further by Carpenter, Kenward, and Vansteelandt (2005) and Kang and Schafer (2007).”
Summary and discussion of “ The central role of the propensity score in observational studies for causal effects (2014, methodology): “25, where x[2] denotes the second covariate.”
Doubly robust uniform confidence band for the conditional average treatment effect function (2016, methodology): “For example, the inverse probability weighting regression adjustment estimator (Kang and Schafer, 2007; Wooldridge, 2007, 2010) is widely known and has been implemented in statistical software packages.”
Contrasting treatment-specific survival using double-robust estimators (2012, neutral): “Note that Kang and Schafer [20] provided an exposition of double-robust estimators in the context of continuous outcomes.”
Robust Inference on Average Treatment Effects with Possibly More Covariates than Observations (2013, positive): “In a sequence of papers, Leeb and Pötscher (2005, 2008a, 2008b, 2009, 2009) have shown that inference relying too heavily on model selection can not be made uniformly valid. Loosely speaking, uniform validity of a confidence interval captures the idea that the interval should have the same quality (”
Propensity score matching and subclassification in observational studies with multi‐level treatments (2015, neutral): “In the binary treatment and missing data case, previous simulations have found that weighting estimators can have high variability, e.g., Kang and Schafer (2007) and Guo and Fraser (2010), especially if the probabilities are close to zero.”