On the Foundations of Noise-free Selective Classification

By Ran El-Yaniv, Yair Wiener. 2010

522 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, background): “This ability can be viewed as a selective prediction of using evidence (or prediction with a reject option) (El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2017, 2019; Kamath et al., 2020).”
Selective Classification for Deep Neural Networks (2017, methodology): “The entire performance proﬁle of such a classiﬁer can be speciﬁed by its risk-coverage curve, deﬁned to be risk as a function of coverage [5].”
To Trust Or Not To Trust A Classifier (2018, background): “Note that the interplay between classiﬁcation rate and reject rate is studied in many various forms e.g. [7, 13, 19, 48, 52, 18, 34, 14, 56, 51].”
Learning with Rejection (2016, methodology): “Using a different approach based on active learning, El-Yaniv and Wiener [11] studied the trade-off between the coverage and accuracy of classifiers and, in a subsequent paper ([12]) provided a strategy to learn a certain type of selective classification, which they define as weakly optimal, that ha”
SSD: A Unified Framework for Self-Supervised Outlier Detection (2021, background): “…et al., 2017) or Variational auto-encoders (Abati et al., 2019; An & Cho, 2015) 2) Classiﬁcation based, such as Deep-SVDD (Ruff et al., 2018; El-Yaniv & Wiener, 2010; Geifman & El-Yaniv, 2017) and 3) Probabilistic detectors, such as density models like Glow and PixelCNN++ (Ren et al.,…”
Addressing Failure Prediction by Learning Model Confidence (2019, background): “Risk-coverage curves [8, 11] depicting the performance of ConfidNet and other baselines for CIFAR10 and SVHN datasets appear in Figure 3.”
SelectiveNet: A Deep Neural Network with an Integrated Reject Option (2019, background): “The entire performance proﬁle of such a model can be speciﬁed by its risk–coverage curve, deﬁned to be the risk as a function of coverage (El-Yaniv & Wiener, 2010; Wiener & El-Yaniv, 2012).”
Selective Question Answering under Domain Shift (2020, background): “This general goal motivates the setting of selective prediction, in which a model outputs both a prediction and a scalar conﬁdence, and abstains on inputs where its conﬁdence is low (El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2017).”
Selective Classification Can Magnify Disparities Across Groups (2020, methodology): “On the theoretical side, early work characterized optimal abstention rules given well-specified models (Chow, 1970; Hellman & Raviv, 1970), with more recent work on learning with perfect precision (El-Yaniv & Wiener, 2010; Khani et al., 2016) and guaranteed risk (Geifman & El-Yaniv, 2017).”
Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition (2012, background): “Also related is classification with reject options, which grants binary classifiers an option to abstain, for a particular cost [32, 10].”
A compression technique for analyzing disagreement-based active learning (2014, methodology): “Moreover, the new method signiﬁcantly simpliﬁes the recent technique of Wiener (2013); El-Yaniv and Wiener (2012, 2010) by completely eliminating the need for the characterizing set complexity measure.”
Theory of Disagreement-Based Active Learning (2014, methodology): “El-Yaniv and Wiener [2010] study an extreme form of selective classification, which they call perfect selective classification, in which the algorithm is required to produce a pair (f, g) with the property that, for every PXY in the realizable case, R(f, g) = 0 (with certainty): that is, whenever th”
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples (2019, methodology): “To move towards a robust open-world deep learning system, we take inspiration from previous work on selective prediction [23, 37, 43, 85] which augments classifiers for in-distribution data with an additional class (referred to as background class) so they can be extended to open-world learning envi”
Agnostic Selective Classification (2011, background): “We first show that the concept of “perfect classification” that was introduced for the realizable case in [1], can be extended to the agnostic setting.”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “In particular, this task jointly assesses a model’s predictive performance and quality of uncertainty estimates (El-Yaniv & Wiener, 2010).”