Evidence on the deleterious impact of sustained use of polynomial regression on causal inference

By Andrew Gelman, Adam Zelizer. 2015

48 citations recorded by the source corpus.

It is common in regression discontinuity analysis to control for third- or fifth-degree polynomials of the assignment variable. Such models can overfit, leading to causal inferences that are substantively implausible and that arbitrarily attribute variation to the high-degree polynomial or the discontinuity. This paper examines two recent studies that make use of regression discontinuity to discuss evident practical problems with these estimates and how they interact with pathologies of the current system of scientific publication. First, we discuss a recent study that estimates the effect on air pollution and life expectancy of a coal-heating policy in China. The reported effects, based on a third-degree polynomial, are statistically significant but substantively dubious, and are sensitive to model choice. This study is indicative of a category of policy analyses where strong claims are based on weak data and methodologies which permit the researcher wide latitude in presenting estimated treatment effects. We then replicate a procedure from Green et al., in which regression discontinuity is used to recover estimated treatment effects relative to an experimental benchmark, to illustrate one practical problem with the regression discontinuity estimates in the coal-heating paper: high-degree polynomials yield noisy estimates of treatment effects that do not accurately convey uncertainty. We recommend that (a) researchers consider the problems which may result from controlling for higher-order polynomials; and (b) that journals recognize that quantitative analyses of policy issues are often inconclusive and relax the implicit rule under which statistical significance is a condition for publication.

Subfields: Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/80211f9eaa0b79523c172f6c4811ded0597a4924

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs (2014, background): “…the outcome is very strong, so that even when a high-degree polynomial is fit to data, that fit is smooth and monotonic, in which case such polynomials can do less damage than they can in examples where the forcing variable is not a good predictor at all (as discussed in Gelman and Zelizer 2015).”
Beyond subjective and objective in statistics (2015, background): “…are subtle Instead of pointing at speciﬁc bad examples, here is a list of some common problems (discussed, for example, in Gelman, 2015, and Gelman and Zelizer, 2015), where we believe that exercising one or more of our listed virtues would improve matters: • Presenting analyses that are…”