Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique

By W. Lin. 2012

674 citations recorded by the source corpus.

Freedman [Adv. in Appl. Math. 40 (2008) 180-193; Ann. Appl. Stat. 2 (2008) 176-196] critiqued ordinary least squares regression adjustment of estimated treatment effects in randomized experiments, using Neyman's model for randomization inference. Contrary to conventional wisdom, he argued that adjustment can lead to worsened asymptotic precision, invalid measures of precision, and small-sample bias. This paper shows that in sufficiently large samples, those problems are either minor or easily fixed. OLS adjustment cannot hurt asymptotic precision when a full set of treatment-covariate interactions is included. Asymptotically valid confidence intervals can be constructed with the Huber-White sandwich standard error estimator. Checks on the asymptotic approximations are illustrated with data from Angrist, Lang, and Oreopoulos's [Am. Econ. J.: Appl. Econ. 1:1 (2009) 136--163] evaluation of strategies to improve college students' achievement. The strongest reasons to support Freedman's preference for unadjusted estimates are transparency and the dangers of specification search.

Subfields: Economics and Econometrics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

In Statistics and Probability

Citations in the corpus

Metalearners for estimating heterogeneous treatment effects using machine learning (2017, background): “The second setup (complex non–linear) is motivated by Wager and Athey (2015). Here the response function are non–linear functions:”
The Econometrics of Randomized Experiments (2016, background): “See for additional discussions Lesaffre and Senn (2003), Samii and Aronow (2012), Rosenbaum (2002), Lin (2013), Schochet (2010), Young (2016), Senn (1994).”
The State of Applied Econometrics - Causality and Policy Evaluation (2016, neutral): “However, this review will be brief, and incomplete, because this continues to be a very active area, with work ranging from econometrics (Manski [1993]) to economic theory (Jackson [2010]). In general, the questions in this literature focus on causal effects in settings where units, often individual”
Heterogeneous Treatment Effects in Digital Experimentation (2014, methodology): “The approximation (31) further ignores uncertainty about µ , and as mentioned in Section 5.4 its variance is based on the same HC covariance advocated in Lin (2013). week”
From Local to Global: External Validity in a Fertility Natural Experiment (2015, result): “In a recent study that comes closer to what we do here, Orr et al. (2017) work with the results of multisite education experiments, using a leaveone-out approach to examine the out-of-sample predictive accuracy of multilevel regression models that model treatment effects as functions of context-leve”
Adjusting treatment effect estimates by post‐stratification in randomized experiments (2013, background): “Lin (2012) extended Freedman’s results and showed that when a treatment-by-covariate interaction is included in the regression, adjustment cannot increase the asymptotic variance.”
SHARP BOUNDS ON THE VARIANCE IN RANDOMIZED EXPERIMENTS (2014, background): “LEE The estimate V̂ a n is also produced by common estimators that presuppose sampling from an infinite superpopulation, including heteroskedasticityrobust variance estimators [12, 16] and the standard variance estimate for mean differences between independent groups [13].”
A paradox from randomization-based causal inference (2014, methodology): “Schochet (2010), Samii and Aronow (2012), Lin (2013) and Imbens and Rubin (2015) pointed out that we can solve this problem by using the Huber– White heteroskedasticity-robust variance estimator (Huber, 1967; White, 1980), and the corresponding Wald test is asymptotically the same as Neyman’s test.”
Optimal a priori balance in the design of controlled experiments (2013, background): “Although carefully conducted post hoc regression adjustments need only to introduce slight bias in misspecified settings (Lin, 2013), as we shall see, optimal a priori balance can achieve the same efficiency with zero finite sample bias.”
Cluster–Robust Variance Estimation for Dyadic Data (2013, positive): “We work within the “agnostic” (Angrist and Imbens, 2002; Lin, 2013) or “assumption lean” (Buja et al., 2014) regression framework developed by Angrist and Pischke (2009), Goldberger (1991), and White (1980 b , 1982).”
Lasso adjustments of treatment effect estimates in randomized experiments (2015, methodology): “It is worth noting that when two diﬀerent adjustments are made for the treatment and control groups as in [7] and here, the covariates do not have to be the same for the two groups.”
Optimality of Matched-Pair Designs in Randomized Controlled Trials (2022, methodology): “Section D of the supplement contains the results for the regression-adjusted estimator in Lin (2013), which additionally includes the interactions of treatment status and covariates.”
Regression‐assisted inference for the average treatment effect in paired experiments (2016, background): “Freedman (2008) and Lin (2013) investigate the impact of regression adjustment on inference for the sample average treatment effect in completely randomized experiments.”
Randomization Tests for Weak Null Hypotheses in Randomized Experiments (2018, background): “k null hypotheses and have power under corresponding alternative hypotheses. Our theory ignores covariates. The analysis of covariance is a classical topic (Fisher 1935) and still attracts attention (Lin 2013; Lu 2016a; Fogarty 2018b,a; Middleton 2018). Tukey (1993) and Rosenbaum (2002b) discussed s”
How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It (2018, background): “Though it is not necessary to do so (randomization eliminates omitted variable bias in expectation), regression adjustment for covariates has been shown to induce only minor bias and to potentially increase e ciency under realistic conditions (e.g., Lin 2013).”