SelectiveNet: A Deep Neural Network with an Integrated Reject Option

By Yonatan Geifman, Ran El-Yaniv. 2019

469 citations recorded by the source corpus.

We consider the problem of selective prediction (also known as reject option) in deep neural networks, and introduce SelectiveNet, a deep neural architecture with an integrated reject option. Existing rejection mechanisms are based mostly on a threshold over the prediction confidence of a pre-trained network. In contrast, SelectiveNet is trained to optimize both classification (or regression) and rejection simultaneously, end-to-end. The result is a deep neural network that is optimized over the covered domain. In our experiments, we show a consistently improved risk-coverage trade-off over several well-known classification and regression datasets, thus reaching new state-of-the-art results for deep selective classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Improving alignment of dialogue agents via targeted human judgements (2022, background): “This ability can be viewed as a selective prediction of using evidence (or prediction with a reject option) (El-Yaniv and Wiener, 2010; Geifman and El-Yaniv, 2017, 2019; Kamath et al., 2020).”
Teaching language models to support answers with verified quotes (2022, background): “This setup could be described as “selective prediction” (also known as prediction with a reject option) (El-Yaniv et al., 2010; Geifman and El-Yaniv, 2017, 2019; Kamath et al., 2020).”
Energy-based Out-of-distribution Detection (2020, methodology): “Auxiliary data allows the model to be explicitly regularized through ﬁne-tuning, producing lower conﬁdence on anomalous examples [2, 9, 26, 27, 37].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…(Cordella et al., 1995; Geifman and El-Yaniv, 2017), to methods involving ensembles of models (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Geifman et al., 2018) or jointly learning to abstain and classify (Bartlett and Wegkamp, 2008; Geifman and El-Yaniv, 2019; Feng et al., 2019).”
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings (2022, methodology): “Selective Prediction (SP) has been studied in machine learning (Chow, 1957; El-Yaniv et al., 2010) and computer vision (Geifman and El-Yaniv, 2017, 2019), but has only recently gained attention in NLP. Kamath et al. (2020) proposed a post-hoc calibration-based SP technique for Question-Answering…”
Out-of-distribution Detection with Deep Nearest Neighbors (2022, background): “…addressed OOD detection by training-time regularization (Lee et al., 2017; Bevandi ´ c et al., 2018; Malinin & Gales, 2018; Hendrycks et al., 2019; Geifman & El-Yaniv, 2019; Hein et al., 2019; Meinke & Hein, 2019; Mohseni et al., 2020; Liu et al., 2020; Jeong & Kim, 2020; Van Amersfoort et al.,…”
Solving Mixed Integer Programs Using Neural Networks (2020, methodology): “We use the SelectiveNet approach (Geifman and El-Yaniv 2019) to train an additional binary classifier that decides which variables to predict a value for and which to refrain from predicting, and optimizes for “coverage” among variables, defined as the ratio of the number of variables predicted vs n”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, methodology): “OOD Detection with Model Fine-tuning An orthogonal line of work explores training with auxiliary outlier data for model regularization [3, 11, 33, 35, 43, 31].”
Mitigating Neural Network Overconfidence with Logit Normalization (2022, background): “…address the out-of-distribution detection problem by training-time regularization (Lee et al., 2017; Bevandi ´ c et al., 2018; Hendrycks et al., 2019; Geifman & El-Yaniv, 2019; Malinin & Gales, 2018; Mohseni et al., 2020; Jeong & Kim, 2020; Liu et al., 2020; Chen et al., 2021; Wei et al., 2021;…”
Consistent Estimators for Learning to Defer to an Expert (2020, methodology): “Approaches range from deferring based on conﬁdence scores [GEY17], learning a deep network with two heads, one for predicting and the other for deferring [GEY19] and learning with portfolio theory inspired loss functions [ZWL + 19].”
Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier (2020, methodology): “, avoiding the harder decisions that are more error-prone. The trade-o between accuracy and ne-grained labeling is controlled by adjusting . The condence score s() can be implemented in various ways [11,25,57]. While RTC is compatible with any of these, we adopt the popular maximum posterior probabi”
Selective Classification Can Magnify Disparities Across Groups (2020, methodology): “Others incorporate abstention into model training (Bartlett & Wegkamp, 2008; Geifman & El-Yaniv, 2019; Feng et al., 2019) and learn to abstain on examples human experts are more likely to get correct (Raghu et al., 2019; Mozannar & Sontag, 2020; De et al., 2020).”