Knowing what you know: valid confidence sets in multiclass and multilabel prediction

By Maxime Cauchois, Suyash Gupta, John C. Duchi. 2020

14 citations recorded by the source corpus.

We develop conformal prediction methods for constructing valid predictive confidence sets in multiclass and multilabel problems without assumptions on the data generating distribution. A challenge here is that typical conformal prediction methods---which give marginal validity (coverage) guarantees---provide uneven coverage, in that they address easy examples at the expense of essentially ignoring difficult examples. By leveraging ideas from quantile regression, we build methods that always guarantee correct coverage but additionally provide (asymptotically optimal) conditional coverage for both multiclass and multilabel prediction problems. To address the potential challenge of exponentially large confidence sets in multilabel prediction, we build tree-structured classifiers that efficiently account for interactions between labels. Our methods can be bolted on top of any classification model---neural network, random forest, boosted tree---to guarantee its validity. We also provide an empirical evaluation suggesting the more robust coverage of our confidence sets.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. E.J. Candes
  2. Matteo Sesia
  3. Yaniv Romano
  4. Leying Guan
  5. Anastasios Nikolas Angelopoulos
  6. Stephen Bates
  7. John C. Duchi
  8. Maxime Cauchois

In Evaluation & Benchmarks

  1. Matteo Sesia
  2. E.J. Candes
  3. Yaniv Romano

In Probabilistic & Bayesian Methods

  1. Matteo Sesia
  2. E.J. Candes
  3. Yaniv Romano

Citations in the corpus