The limits of distribution-free conditional predictive inference

By Rina Foygel Barber, E.J. Candes, Aaditya Ramdas, Robert Tibshirani. Information and Inference A Journal of the IMA, 2019

376 citations recorded by the source corpus.

We consider the problem of distribution-free predictive inference, with the goal of producing predictive coverage guarantees that hold conditionally rather than marginally. Existing methods such as conformal prediction offer marginal coverage guarantees, where predictive coverage holds on average over all possible test points, but this is not sufficient for many practical applications where we would like to know that our predictions are valid for a given individual, not merely on average over a population. On the other hand, exact conditional inference guarantees are known to be impossible without imposing assumptions on the underlying distribution. In this work, we aim to explore the space in between these two and examine what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/ebc3193ae46286f82f741143f5d67891c1625209

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction (2020, background): “Our work, then, is a stepping stone toward more uniform notions of validity, and we believe exploring approaches to this—perhaps by distributionally robust optimization (Delage and Ye, 2010; Duchi and Namkoong, 2018), perhaps by uniform convergence arguments (Barber et al., 2019, Sec.”
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, background): “This group also did fundamental work circumscribing the conditions under which distribution-free conditional guarantees can exist [69].”
Conformal Risk Control (2022, background): “…algorithm, mainly targeting deviations from exchangeability (Tibshirani et al., 2019; Gibbs & Candes, 2021; Barber et al., 2022; Fannjiang et al., 2022) and improved conditional coverage (Barber et al., 2020; Romano et al., 2019; Guan, 2020; Romano et al., 2020; Angelopoulos et al., 2021b).”
Conformal Prediction Under Covariate Shift (2019, background): “Combined with our somewhat pessimistic results in Barber et al. (2019), many important practical and philosophical problems in assumption-lean conditional predictive inference remain open.”
Classification with Valid and Adaptive Coverage (2020, background): “assumptions [1, 23], which we are not willing to make in this paper.”
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction (2021, methodology): “Lemma A.1 describes the elementary relationship used in the proof from previous work on weighted conformal prediction (Barber et al. 2019 a ), and we state it here for the reader’s convenience.”
Conformal inference of counterfactuals and individual treatment effects (2020, background): “Without modelling assumptions, it is known to be impossible to construct non- trivial prediction intervals with guaranteed conditional coverage (Barber et al., 2019a).”
With Malice Towards None: Assessing Uncertainty via Equalized Coverage (2019, background): “Our contribution is nevertheless a little outside of classical conformal inference as we seek a form of conditional rather than marginal coverage guarantee [25–27].”
Knowing what you know: valid confidence sets in multiclass and multilabel prediction (2020, background): “We sketch a derivation of inequality (22); see also [1, Theorem 5] for related arguments.”
Adaptive Conformal Inference Under Distribution Shift (2021, methodology): “Perhaps the most powerful and ﬂexible tools for solving this problem come from conformal inference [see e.g. 34, 16, 32, 22, 31, 15, 3] .”
Conformal prediction with conditional guarantees (2023, neutral): “Finally, in this context we have which is a tractable function.”
Distribution-Free, Risk-Controlling Prediction Sets (2021, background): “…prediction is a generic approach, and much recent work has focused on designing speciﬁc conformal procedures to have good performance according to metrics such as small set sizes [14], approximate coverage in all regions of feature space [15–21], and errors balanced across classes [14, 22–24].”
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods (2019, background): “Roughly speaking, in the case of marginal coverage, one averages over both XN+1 and YN+1 (i.e., the probability P refers to a joint measure on X × Y), while in the case of conditional coverage, XN+1 is fixed and the average in taken over YN+1 only (Barber et al., 2020).”