Conformal Prediction Under Covariate Shift

By Robert Tibshirani, Rina Foygel Barber, E.J. Candes, Aaditya Ramdas. Neural Information Processing Systems, 2019

742 citations recorded by the source corpus.

We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known---or, in practice, can be estimated accurately with access to a large set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more generally, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal methodology, including latent variable and missing data problems.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/f08e13d65cb17856427b429d79f01922584a6f01

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Conformal inference of counterfactuals and individual treatment effects (2020, methodology): “The method is implemented in the grf package (Tibshirani et al., 2019a).”
Adaptive Conformal Inference Under Distribution Shift (2021, methodology): “[33] showed that valid prediction sets can be obtained by re-weighting the calibration data using the likelihood ratio between P1 and P0.”
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction (2021, methodology): “Barber et al. (2019 a ) introduce a diﬀerent notion of approximate conditional coverage, where instead of ﬁnding C ( x 0 ) that achieves conditional coverage of Y n +1 given X n +1 = x 0 , the authors consider C ( x 0 ) that covers ˜ Y whose feature value ˜ X is distributed according to some…”
Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, background): “In a similar spirit, Tibshirani et al. (2019) suggest a notion of local conditional coverage, where one asks for coverage in a neighborhood of each point, weighted according to a chosen kernel.”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): “This is also a well-known result; we prove it here for completeness (and see Tibshirani et al. (2019) for an identical proof).”
PAC Confidence Predictions for Deep Neural Network Classifiers (2020, background): “, 2020b) or uncertainty set prediction with covariate shift (Cauchois et al., 2020; Tibshirani et al., 2019).”
Second opinion needed: communicating uncertainty in medical machine learning (2021, positive): “However, recent work by Barber et al. extends conformal inference guarantees to the setting of covariate shift 28 .”
Distribution-Free, Risk-Controlling Prediction Sets (2021, background): “Further extensions of conformal prediction address topics such as distribution estimation [25], causal inference [26], and handling or testing distribution shift [27–29].”
Conformal Risk Control (2022, methodology): “Recently there have been many extensions of the conformal algorithm, mainly targeting deviations from exchangeability (Tibshirani et al., 2019; Gibbs & Candes, 2021; Barber et al., 2022; Fannjiang et al., 2022) and improved conditional coverage (Barber et al., 2020; Romano et al., 2019; Guan, 2020;…”
Conformal prediction with conditional guarantees (2023, positive): “There have been a number of previous works in this area that also establish coverage under a single covariate shift that is either known a-priori (Tibshirani et al. (2019)), or estimated from unlabeled data (Qiu et al. (2022), Yang et al. (2022)).”