Distribution-Free Predictive Inference for Regression

By Jing Lei, M. G'Sell, Alessandro Rinaldo, Robert Tibshirani, Larry A. Wasserman. 2016

1,299 citations recorded by the source corpus.

ABSTRACT We develop a general framework for distribution-free predictive inference in regression, using conformal inference. The proposed methodology allows for the construction of a prediction band for the response variable using any estimator of the regression function. The resulting prediction band preserves the consistency properties of the original estimator under standard assumptions, while guaranteeing finite-sample marginal coverage even when these assumptions do not hold. We analyze and compare, both empirically and theoretically, the two major variants of our conformal framework: full conformal inference and split conformal inference, along with a related jackknife method. These methods offer different tradeoffs between statistical accuracy (length of resulting prediction intervals) and computational efficiency. As extensions, we develop a method for constructing valid in-sample prediction intervals called rank-one-out conformal inference, which has essentially the same computational efficiency as split conformal inference. We also describe an extension of our procedures for producing prediction bands with locally varying length, to adapt to heteroscedasticity in the data. Finally, we propose a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference. Accompanying this article is an R package conformalInference that implements all of the proposals we have introduced. In the spirit of reproducibility, all of our empirical results can also be easily (re)generated using this package.

Subfields: Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

Citations in the corpus

Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI) (2018, background): “LOCO [86] use a local feature importance as well. d: EXAMPLE-BASED EXPLANATION Example-based explanation techniques select particular instances of the dataset to explain the behavior of machine learning models.”
Visualizing the Feature Importance for Black Box Models (2018, methodology): “Another approach to measure the feature importance in a model-agnostic fashion is the leave-one-covariate-out (LOCO) method [13] which requires to refit the model as the approach is based on leaving features out instead of permuting them.”
Verified Uncertainty Calibration (2019, methodology): “Besides calibration, other ways of producing and quantifying uncertainties include Bayesian methods [40] and conformal prediction [41, 42].”
Cross-Validation With Confidence (2017, methodology): “[20] describe a sample splitting method for assessing variable importance in building predictive regression models, which also uses a hypothesis testing idea.”
Deterministic Local Interpretable Model-Agnostic Explanations for Stable Explainability (2021, neutral): “In the same vein, Leave-One-Covariate-Out (LOCO) [12] and Local Rule-based Explanation (LORE) [13] are other popular techniques for generating local explanation models that offer local variable importance measures.”
Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges (2020, background): “re importance [28,16] is a popular importance measure, originally suggested for random forests [13]. Some importance measures rely on removing features from the training data and retraining the model [65]. An alternative are variance-based measures [40]. See [125] for an overview of importance measu”
: "Statistical Modeling: The Two Cultures" (Statistical Science, 2001, 16(3), 199-231) (neutral): “Recent interests have led to building inferential (statistical) tools around algorithm-driven learning, see, e.g., works on conformal inference by Wasserman et al. (2020); Lei et al. (2018); Barber et al. (2021) for example.”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences (2021, result): “Why (a given prediction) •Describe how features of the instance, or what key features, determine the model’s prediction of it∗ • Or describe rules that the instance fits to guarantee the prediction† • Or show similar examples with the same predicted outcome to justify the model’s prediction‡ LIME∗ [”
Conformal Prediction Under Covariate Shift (2019, methodology): “Moreover, we refer to Lei and Wasserman (2014); Lei et al. (2018) for recent developments in the areas of nonparametric and high-dimensional regression.”
Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance (2020, methodology): “1 Bad Tests or Bad Interpretations? Given these results, one may be tempted to conclude that tests of the style proposed in work such as Mentch and Hooker (2016), Lei et al.”
A comparison of some conformal quantile regression methods (2019, methodology): “In general, the conformal prediction intervals described in Lei et al. (2018) will not converge to those of our oracle if 3 This was first suggested by Yaniv Romano through personal communication. the noise is heteroscedastic, regardless of the consistency of the black box regression estimator ?”
Understanding Global Feature Contributions With Additive Importance Measures (2020, methodology): “The canonical method for this is a feature ablation study where, in addition to a model f trained on all features, separate models f 1 , f 2 , . . . , f d are trained with individual features excluded [2, 15, 20].”
Statistical stability indices for LIME: Obtaining reliable explanations for machine learning models (2020, methodology): “The idea has been ﬁrst introduced by Breiman (Breiman, 2001) for the Random Forest model and has been generalised to a model-agnostic framework, named LOCO (Lei, G’Sell, Rinaldo, Tibshirani, & Wasserman, 2018).”
Explaining by Removing: A Unified Framework for Model Explanation (2020, methodology): “…to how Occlusion, PredDiff and CXPlain calculate local feature importance and how permutation tests and feature ablation (LOCO) summarize global feature importance (Breiman, 2001; Strobl et al., 2008; Zeiler and Fergus, 2014; Zintgraf et al., 2017; Lei et al., 2018; Schwab and Karlen, 2019).”
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction (2021, methodology): “) are more homogeneous across diﬀerent x ∈ R p (Lei & Wasserman 2014, Izbicki et al. 2019, Lei et al. 2018, Romano et al. 2019, Gupta et al. 2021).”