A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

By Anastasios Nikolas Angelopoulos, Stephen Bates. arXiv.org, 2021

1,169 citations recorded by the source corpus.

Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c3ea8eb80bc8ca0b21efa273b9e4a9fd059c65be

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Ziwei Liu
5. Kaiyang Zhou
6. Dara Bahri
7. Jingkang Yang
8. Tal Schuster

Citations in the corpus

• Confident Adaptive Language Modeling (2022, methodology): “Our calibration procedure for connecting global constraints to local decisions, relates to recent research around distribution-free uncertainty quantification [1; 62; 77].”
• Generalized Out-of-Distribution Detection: A Survey (2021, positive): “Their innovative approach culminates in a novel OOD detection method that surpasses existing techniques in both theoretical robustness and empirical performance.”
• Conformal Risk Control (2022, background): “See Angelopoulos & Bates (2021) for a modern introduction to the area or Shafer & Vovk (2008) for a more classical alternative.”