A tutorial on conformal prediction

By Glenn Shafer, Vladimir G Vovk. 2007

1,695 citations recorded by the source corpus.

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability e, together with a method that makes a prediction ŷ of a label y, it produces a set of labels, typically containing ŷ, that also contains y with probability 1 e. Conformal prediction can be applied to any method for producing ŷ: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1 e of the time, even though they are based on an accumulating data set rather than on independent data sets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in Algorithmic Learning in a Random World, by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

Subfields: Artificial Intelligence, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical Learning Theory

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, methodology): “We now formulate the main contribution of this paper, which is a distribution-free and model-agnostic method based on CP for guaranteeing any performance bound an end-user chooses to specify [36].”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, methodology): “To formalize this, we operate in the framework of conformal prediction [24] and compute both the confidence and credibility of DkNN predictions.”
Deep Weighted Averaging Classifiers (2018, background): “Conformal methods refer to a broad set of ideas which aim to provide theoretical guarantees on error rates in classification or regression [26, 45, 47, 54].”
DeltaGrad: Rapid retraining of machine learning models (2020, methodology): “Techniques based on conformal prediction (e.g., Shafer & Vovk, 2008) rely on retraining models on subsets of the data.”
Verified Uncertainty Calibration (2019, methodology): “Besides calibration, other ways of producing and quantifying uncertainties include Bayesian methods [40] and conformal prediction [41, 42].”
Kernel Ridge Regression (2013, methodology): “The method was further developed in [17]; see [13] for a tutorial.”
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (2021, background): “For some specific score functions, the set in (16) can actually be computed exactly even for continuous Y , and we refer the reader to [1] and [38] for a summary of such cases and [39, 40] for recent developments.”
Second opinion needed: communicating uncertainty in medical machine learning (2021, methodology): “However, with an approach known as conformal inference 26 , it ’ s possible to obtain exact marginal coverage guarantees per sample for virtually any standard machine learning model in both regression and classi ﬁ cation settings.”
Conformal Prediction Under Covariate Shift (2019, methodology): “Conformal prediction , a framework pioneered by Vladimir Vovk and colleagues in the 1990s, provides a means for achieving this goal, relying only on exchangeablility of the training and test data.”
A unifying framework for computational reinforcement learning theory (2009, neutral): “, Sontag [1998]).”
The limits of distribution-free conditional predictive inference (2019, background): “As another example, Vovk [2012], Lei and Wasserman [2014] consider a version of conformal prediction that guarantees coverage within each one of a finite number of subgroups, i.”
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction (2021, background): “…+1 , Y n +1 ) as the ( n + 1) th observation, CP achieves (1.1) under the assumption that Z n +1 is also independently generated from P XY , without additional distributional assumptions on P XY itself (Vovk et al. 2005, Shafer & Vovk 2008, Vovk et al. 2009, Lei & Wasserman 2014, Lei et al. 2018).”
Conformal Risk Control (2022, background): “See Angelopoulos & Bates (2021) for a modern introduction to the area or Shafer & Vovk (2008) for a more classical alternative.”
Adaptive Conformal Inference Under Distribution Shift (2021, methodology): “Perhaps the most powerful and ﬂexible tools for solving this problem come from conformal inference [see e.g. 34, 16, 32, 22, 31, 15, 3] .”
Interpreting Neural Networks with Nearest Neighbors (2018, background): “Conformity follows from the framework of conformal prediction (Shafer and Vovk, 2008) and estimates how much the training data supports a classification decision.”
Transcend: Detecting Concept Drift in Malware Classification Models (2017, background): “Comparison with Conformal Predictor.”