Predictive inference with the jackknife+

By Rina Foygel Barber, E.J. Candes, Aaditya Ramdas, Robert Tibshirani. Annals of Statistics, 2019

442 citations recorded by the source corpus.

This paper introduces the jackknife+, which is a novel method for constructing predictive confidence intervals. Whereas the jackknife outputs an interval centered at the predicted response of a test point, with the width of the interval determined by the quantiles of leave-one-out residuals, the jackknife+ also uses the leave-one-out predictions at the test point to account for the variability in the fitted regression function. Assuming exchangeable training samples, we prove that this crucial modification permits rigorous coverage guarantees regardless of the distribution of the data points, for any algorithm that treats the training points symmetrically. Such guarantees are not possible for the original jackknife and we demonstrate examples where the coverage rate may actually vanish. Our theoretical and empirical analysis reveals that the jackknife and the jackknife+ intervals achieve nearly exact coverage and have similar lengths whenever the fitting algorithm obeys some form of stability. Further, we extend the jackknife+ to K-fold cross validation and similarly establish rigorous coverage properties. Our methods are related to cross-conformal prediction proposed by Vovk [2015] and we discuss connections.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/1896136f931aa080599e1c1314d48bccbaa14e81

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Adaptive Conformal Inference Under Distribution Shift (2021, methodology): “Perhaps the most powerful and ﬂexible tools for solving this problem come from conformal inference [see e.g. 34, 16, 32, 22, 31, 15, 3] .”
Conformal inference of counterfactuals and individual treatment effects (2020, background): “Without modelling assumptions, it is known to be impossible to construct non- trivial prediction intervals with guaranteed conditional coverage (Barber et al., 2019a).”
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, methodology): “In particular, cross-conformal prediction [41] and CV+/Jackknife+ [42] both use a small number of model fits, but still use all data for both model fitting and calibration.”
Conformal prediction beyond exchangeability (2022, methodology): “B.1 Background The jackknife+ [Barber et al., 2021] (closely related to “cross-conformal prediction” [Vovk, 2015]) is a method that offers a compromise between the computational and statistical costs of the split and full conformal methods.”
: "Statistical Modeling: The Two Cultures" (Statistical Science, 2001, 16(3), 199-231) (unknown): “Recent interests have led to building inferential (statistical) tools around algorithm-driven learning, see, e.g., works on conformal inference by Wasserman et al. (2020); Lei et al. (2018); Barber et al. (2021) for example.”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, background): “Recent results have shown that carefully designed jackknife estimators [5] can achieve worst-case coverage guarantees on regression problems.”