With Malice Towards None: Assessing Uncertainty via Equalized Coverage

By Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, E.J. Candes. 2019

67 citations recorded by the source corpus.

An important factor to guarantee a fair use of data-driven recommendation systems is that we should be able to communicate their uncertainty to decision makers. This can be accomplished by constructing prediction intervals, which provide an intuitive measure of the limits of predictive performance. To support equitable treatment, we force the construction of such intervals to be unbiased in the sense that their coverage must be equal across all protected groups of interest. We present an operational methodology that achieves this goal by offering rigorous distribution-free coverage guarantees holding in finite samples. Our methodology, equalized coverage, is flexible as it can be viewed as a wrapper around any predictive algorithm. We test the applicability of the proposed framework on real data, demonstrating that equalized coverage constructs unbiased prediction intervals, unlike competitive methods.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Consistent Accelerated Inference via Confident Adaptive Transformers (2021, background): “Our work adds to several recent directions that explore CP in the context of risk-mitigating applications (Lei and Candès, 2020; Romano et al., 2020; Bates et al., 2020; Fisch et al., 2021a, inter alia ), or meta-learning settings (Fisch et al., 2021b).”
Conformal prediction with conditional guarantees (2023, unknown): “In particular, we show how our method can be used to satisfy two popular coverage desiderata: group-conditional coverage and coverage under covariate shift.”
Few-shot Conformal Prediction with Auxiliary Tasks (2021, background): “…conformal prediction in the context of various validity conditions, such as conditional, risk-controlling, admissible, or equalized coverage (Chernozhukov et al., 2019; Cauchois et al., 2020; Kivaranovic et al., 2020; Romano et al., 2019; 2020; Bates et al., 2020; Fisch et al., 2021, inter alia ).”
Classification with Valid and Adaptive Coverage (2020, methodology): “Finally, the approximate conditional coverage that we seek in this work is highly relevant within the broader framework of fairness, as discussed by [17] within a regression setting.”