Classification with Valid and Adaptive Coverage

By Yaniv Romano, Matteo Sesia, E.J. Candes. Neural Information Processing Systems, 2020

528 citations recorded by the source corpus.

Conformal inference, cross-validation+, and the jackknife+ are hold-out methods that can be combined with virtually any machine learning algorithm to construct prediction sets with guaranteed marginal coverage. In this paper, we develop specialized versions of these techniques for categorical and unordered response labels that, in addition to providing marginal coverage, are also fully adaptive to complex data distributions, in the sense that they perform favorably in terms of approximate conditional coverage compared to alternative methods. The heart of our contribution is a novel conformity score, which we explicitly demonstrate to be powerful and intuitive for classification problems, but whose underlying principle is potentially far more general. Experiments on synthetic and real data demonstrate the practical value of our theoretical guarantees, as well as the statistical advantages of the proposed methods over the existing alternatives.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/00215f32433e4e69ddb5a678b3f02568334d67ca

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Conformal prediction with conditional guarantees (2023, unknown): “…step yields a weak marginal coverage guarantee, a good conformity score is crucial to constructing a prediction set with good, even if not provable, conditional validity (Lei and Wasserman (2014), Romano et al. (2019, 2020b), Chernozhukov et al. (2021), Sesia and Romano (2021), Guan (2022)).”
Conformal inference of counterfactuals and individual treatment effects (2020, background): “For instance, with estimates q̂ lo (x) and q̂ hi (x) of the loth and hith conditional quantiles of Y ∣ X = x, the conformal quantile regression (CQR) introduced in Romano et al. (2019) as a variant of the standard conformal inference, would produce an interval estimate of the form”
Conformal Risk Control (2022, background): “…algorithm, mainly targeting deviations from exchangeability (Tibshirani et al., 2019; Gibbs & Candes, 2021; Barber et al., 2022; Fannjiang et al., 2022) and improved conditional coverage (Barber et al., 2020; Romano et al., 2019; Guan, 2020; Romano et al., 2020; Angelopoulos et al., 2021b).”
Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction (2020, methodology): “In work building oﬀ of the initial post of this paper to the arXiv , Romano et al. (2020) construct conformal conﬁdence sets for multi-class problems by leveraging pivotal p -value-like quantities, which provide conditional coverage when models are well-speciﬁed.”
Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, methodology): “Given such a model, the oracle procedure from Romano et al. (2020) would return the correct label with probability 1 − α and the empty set with probability α .”
Adaptive Conformal Inference Under Distribution Shift (2021, unknown): “The generality of this approach has facilitated the development of a large suite of conformal methods, each specialized to a speciﬁc prediction problem of interest [e.g. 30, 11, 23, 8, 24, 21].”