Models as Approximations: How Random Predictors and Model Violations Invalidate Classical Inference in Regression

By Andreas Buja, R. Berk, L. Brown, E. George, E. Pitkin, Mikhail Traskin, K. Zhan, L. Zhao. 2014

33 citations recorded by the source corpus.

We review and interpret the early insights of Halbert White who over thirty years ago inaugurated a form of statistical inference for regression models that is asymptotically correct even under "model misspecification," that is, under the assumption that models are approximations rather than generative truths. This form of inference, which is pervasive in econometrics, relies on the "sandwich estimator" of standard error. Whereas linear models theory in statistics assumes models to be true and predictors to be fixed, White's theory permits models to be approximate and predictors to be random. Careful reading of his work shows that the deepest consequences for statistical inference arise from a synergy --- a "conspiracy" --- of nonlinearity and randomness of the predictors which invalidates the ancillarity argument that justifies conditioning on the predictors when they are random. Unlike the standard error of linear models theory, the sandwich estimator provides asymptotically correct inference in the presence of both nonlinearity and heteroskedasticity. An asymptotic comparison of the two types of standard error shows that discrepancies between them can be of arbitrary magnitude. If there exist discrepancies, standard errors from linear models theory are usually too liberal even though occasionally they can be too conservative as well. A valid alternative to the sandwich estimator is provided by the "pairs bootstrap"; in fact, the sandwich estimator can be shown to be a limiting case of the pairs bootstrap. We conclude by giving meaning to regression slopes when the linear model is an approximation rather than a truth. --- In this review we limit ourselves to linear least squares regression, but many qualitative insights hold for most forms of regression.

Subfields: Applied Mathematics, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Mathematics

In Statistics and Probability

Citations in the corpus

A Hierarchy of Limitations in Machine Learning (2020, unknown): “…1998; Freedman, 2009), interpreting probability (Freedman, 2009; Cox, 1990), and determining what the “inferences” of statistical inference are to (Kass, 2011), especially given model misspecification (Buja et al., 2014); so long as out-of-sample success can be demonstrated, none of that matters.”
Semi-supervised inference: General theory and estimation of means (2016, methodology): “More details of the model can be found in, for example, Buja et al. (2014, 2016).”
Impartial Predictive Modeling: Ensuring Fairness in Arbitrary Models (2016, unknown): “This is crucially important in legal cases and is closely related to inference under model-misspeciﬁcation (Buja et al., 2014).”
Assumption Lean Regression (2018, background): “Table 1 shows that some of the bootstrap and sandwich standard errors are rather different from the conventional standard errors, indicating indirectly that the conditional Poisson model is misspecified (Buja et al. 2016a).”
Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression (2018, background): “Under a possibly misspecified linear model, the estimation of the asymptotic variance of β̂ involves ∑ X2 i (Yi−Xiβ)(2), where β is the limit of β̂; see Buja et al. (2014) for details.”
Distribution-Free Predictive Inference for Regression (2016, background): “This has been studied in, for example, Wasserman (2014); Buja et al. (2014); Tibshirani et al. (2016).”