Conditional validity of inductive conformal predictors

By Vladimir G Vovk. 2012

573 citations recorded by the source corpus.

Conformal predictors are set predictors that are automatically valid in the sense of having coverage probability equal to or exceeding a given confidence level. Inductive conformal predictors are a computationally efficient version of conformal predictors satisfying the same property of validity. However, inductive conformal predictors have only been known to control unconditional coverage probability. This paper explores various versions of conditional validity and various ways to achieve them using inductive conformal predictors and their modifications. In particular, it discusses a convenient expression of one of the modifications in terms of ROC curves.

Subfields: Control and Systems Engineering, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

In Statistics and Probability

Citations in the corpus

Deep Weighted Averaging Classifiers (2018, neutral): “For more details, please refer to [53, 54].”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, methodology): “First, we formalize a PAC-type 2-parameter validity deﬁnition (similar to training conditional CP in Vovk (2012)): Deﬁnition 4.6 ( ( δ, (cid:15) ) task validity) .”
Knowing what you know: valid confidence sets in multiclass and multilabel prediction (2020, background): “Vovk et al. [33] introduce (split-)conformal inference, which splits the ﬁrst n samples of the exchangeable pairs { ( X i , Y i ) } n +1 i =1 into two sets (say, each of size n 1 and n 2 respectively) where the ﬁrst training set ( I 1 ) is used to learn a scoring function s : X × Y → R and the secon”
Predictive inference with the jackknife+ (2019, unknown): “Their work is based on the conformal prediction method (see Vovk, Gammerman and Shafer [29], Lei et al. [14] for background), which provably achieves distribution-free predictive coverage at the target level 1 − α but at an extremely high computational cost.”
Conformal Risk Control (2022, background): “Most relevant to us is recent work on risk control in high probability (Vovk, 2012; Bates et al., 2021; Angelopoulos et al., 2021a) and its applications (Park et al., 2020; Fisch et al., 2022; Schuster et al., 2021; 2022; Sankaranarayanan et al., 2022; An-gelopoulos et al., 2022a;b, inter alia ).”
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction (2021, methodology): “Non-trivial ﬁnite sample and distribution-free conditional coverage is impossible for continuous distributions (Lei & Wasserman 2014, Vovk 2012).”
Cross-conformal predictors (2012, background): “Achieving some of these kinds (such as label and object conditional validity, in the terminology of [20]) requires modifying the definition of conformal predictors.”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, methodology): “Instead, we apply the following dataset-conditional guarantee [9] which is conditioned on a particular calibration dataset being sampled, and thus can be applied to new test data without re-calibration: where Beta − 1 N +1 − v,v ( δ ) denotes the inverse CDF (quantile) level of δ in a Beta…”
Conformal Prediction Under Covariate Shift (2019, background): “As it turns out, maintaining that (17) hold for almost all 6 x 0 ∈ R d and all distributions P is far too strong: Vovk (2012); Lei and Wasserman (2014) prove that any method with such a property must yield an interval (cid:98) C n ( x 0 ) with inﬁnite expected length at any non-atom point 7 x 0 ,…”
Conformal inference of counterfactuals and individual treatment effects (2020, methodology): “Conformal inference was introduced by Vladimir Vovk and his collaborators (e.g. Balasubramanian et al., 2014; Gammerman & Vovk, 2007; Shafer & Vovk, 2008; Vovk, 2012; Vovk, 2013; Vovk, 2015; Vovk et al., 2009).”
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, methodology): “A simple proof of this fact is available in [10].”
Conformal prediction with conditional guarantees (2023, unknown): “Finally, in this context we have which is a tractable function.”
Distribution-Free Predictive Inference for Regression (2016, background): “Also, for a related result, see Corollary 1 of Vovk (2013).”
Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, background): “We say that a set-valued predictor C : Distribution-free guarantees on conditional coverage are impossible (Vovk, 2012; Lei & Wasserman, 2014), but many algorithms try to satisfy it approximately (Romano et al., 2019; 2020; Cauchois et al., 2020).”
Least Ambiguous Set-Valued Classifiers With Bounded Error Levels (2016, methodology): “Theorem 3 of Vovk et al. (2016) provides a characterization, under total coverage, of the optimal classifier among all idealized conformal classifiers (classifiers obtained by thresholding some function g(x, y)).”
The limits of distribution-free conditional predictive inference (2019, methodology): “2 [Rephrased from [13,8]] Suppose that Ĉn satisfies (1 − α)-CC (1.”
Distribution-Free, Risk-Controlling Prediction Sets (2021, background): “Unlike conformal prediction, however, we pursue the high-probability error guarantees of tolerance regions and thus rely on entirely different proof techniques—see [34] for a discussion of their relationship.”