High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
By Tim Pearce, Alexandra Melike Brintrup, Mohamed H. Zaki, Andy Neely. International Conference on Machine Learning, 2018
334 citations recorded by the source corpus.
This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
View Paper: https://www.semanticscholar.org/paper/7a0ba4f62a1d98ad5f1e0bab1bdd78dba9ca8ecc
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Jitendra Malik
- Anastasios Nikolas Angelopoulos
- Michael I. Jordan
- Stephen Bates
- Trevor Darrell
- Dacheng Tao
- Pieter Abbeel
In Statistical and Nonlinear Physics
- David Lopez-Paz
- Jitendra Malik
- Abbas Khosravi
- Mohammad Ghavamzadeh
- Michael I. Jordan
- Xiaochun Cao
- Saeid Nahavandi
- E.J. Candes
Citations in the corpus
- A survey of uncertainty in deep neural networks (2021, methodology): “In order to evaluate the correctness of the predicted intervals the Prediction Interval Coverage Probability (PICP) can be applied [259], [260].”
- Single-Model Uncertainties for Deep Learning (2018, methodology): “QualityDriven [60] is a state-of-the art deep model to estimate prediction intervals, that minimizes a smooth surrogate of the PICP/MPIW metrics.”
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[202] Tabular data Neural Networks Quality-Driven Ensembles (QDEns) LossQD = MPIWcapt.”
- Conformalized Quantile Regression (2019, background): “Recently, new deep learning techniques have been proposed [26–29] for constructing prediction intervals.”
- Uncertainty Sets for Image Classifiers using Conformal Prediction (2020, background): “Lastly, there are alternative approaches to returning prediction sets not based on conformal prediction (Pearce et al., 2018; Zhang et al., 2018).”