Inductive Conformal Prediction: Theory and Application to Neural Networks

By Harris Papadopoulos. 2008

293 citations recorded by the source corpus.

Traditional machine learning algorithms for pattern recognition just output simple predictions, without any associated confidence values. Confidence values are an indication of how likely each prediction is of being correct. In the ideal case, a confidence of 99% or higher for all examples in a set, means that the percentage of erroneous predictions in that set will not exceed 1%. Knowing the likelihood of each prediction enables us to assess the extent to which we can rely on it. For this reason, predictions that are associated with some kind of confidence values are highly desirable in many risk-sensitive applications, such as those used for medical diagnosis or financial analysis. In fact, such information can benefit any application that requires human-computer interaction, as confidence values can be used to determine the way in which each prediction will be treated. For instance, a filtering mechanism can be employed so that only predictions which satisfy a certain level of confidence will be taken into account, while the rest can be discarded or passed on to a human for judgement. There are two main areas in mainstream machine learning that can be used in order to obtain some kind of confidence values; the Bayesian framework and the theory of Probably Approximately Correct learning (PAC theory). Quite often the Bayesian framework is used for producing algorithms that complement individual predictions with probabilistic measures of their quality. On the other hand, PAC theory can be used for producing upper bounds on the probability of error for a given algorithm with respect to some confidence level 1 − δ. Both of these approaches however, have their drawbacks. In order to apply the Bayesian framework one is required to have some prior knowledge about the distribution that generates the data. When the correct prior is known, Bayesian methods provide optimal decisions. For real world data sets though, as the required knowledge is not available, one has to assume the existence of an arbitrarily chosen prior. In this case, if the assumed prior is incorrect, the resulting confidence levels may also be “incorrect”; for example the predictive regions output for the 95% confidence level may contain the true label in much less than 95% of the cases. This signifies a major failure, as we would expect confidence levels to bound the percentage of expected errors. An experimental demonstration of how misleading Bayesian methods can become when their assumptions are violated can be found in (Melluish et al., 2001).

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, methodology): “Thus, we use an inductive variant of conformal prediction [83], [84] that does not require retraining the underlying classifier for each query because it assumes the existence of a calibration set—holdout data that does not overlap with the training or test data.”
Aleatoric and Epistemic Uncertainty in Machine Learning: A Tutorial Introduction (2019, background): “Conformal prediction as outlined above realizes transductive inference, although inductive variants also exist (Papadopoulos, 2008).”
Adaptive Conformal Inference Under Distribution Shift (2021, methodology): “Additionally, we remark that the method outlined above is often referred to as split or inductive conformal inference [27, 34, 26].”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): ““Split” conformal prediction (Papadopoulos, 2008)—which uses a held-out training set to learn S, and therefore also preserves exchangeability—is a more computationally attractive alternative, but comes at the expense of predictive efficiency when data is limited.”
The limits of distribution-free conditional predictive inference (2019, methodology): “The split conformal prediction algorithm, introduced in [10,14] (under the name ‘inductive conformal prediction’) and studied further by [9,13,7], is a well-known method that achieves distribution-free marginal coverage guarantees.”
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods (2019, background): “Conformal prediction as outlined above realizes transductive inference, although inductive variants also exist (Papadopoulos 2008).”
Conformalized Quantile Regression (2019, methodology): “In contrast, the method of split, or inductive, conformal prediction [2, 8] avoids this problem, at the cost of splitting the data.”
A comparison of some conformal quantile regression methods (2019, methodology): “Throughout this paper, we follow the split-conformal approach to conformal inference (Papadopoulos et al., 2002; Papadopoulos, 2008; Lei et al., 2018) adopted in Romano et al. (2019) and Kivaranovic et al. (2019), since it is computationally feasible even with large data.”
Predictive inference with the jackknife+ (2019, unknown): “Papadopoulos [16], Vovk [26], Lei et al.”