Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers

By Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, Theodore L. Willke. 2018

267 citations recorded by the source corpus.

As deep learning methods form a critical part in commercially important applications such as autonomous driving and medical diagnostics, it is important to reliably detect out-of-distribution (OOD) inputs while employing these algorithms. In this work, we propose an OOD detection algorithm which comprises of an ensemble of classifiers. We train each classifier in a self-supervised manner by leaving out a random subset of training data as OOD data and the rest as in-distribution (ID) data. We propose a novel margin-based loss over the softmax output which seeks to maintain at least a margin m between the average entropy of the OOD and in-distribution samples. In conjunction with the standard cross-entropy loss, we minimize the novel loss to train an ensemble of classifiers. We also propose a novel method to combine the outputs of the ensemble of classifiers to obtain OOD detection score and class prediction. Overall, our method convincingly outperforms Hendrycks et al. [7] and the current state-of-the-art ODIN [13] on several OOD detection benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, methodology): “In this subsection, we examine the model robustness against corrupted input [12] and Out-Of-Distribution (OOD) data [35].”
Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model (2022, background): “To alleviate the reliability of DNNs on big data and elaborate annotations, different paradigms such as semisupervised learning [34], few-shot learning [25, 27, 36], zero-shot learning [21,30,33], self-supervised learning [31], open-set learning [6, 22, 29] and advanced training strategies [17, 35] ”
Anomalous Instance Detection in Deep Learning: A Survey (2020, methodology): “In [48], the algorithm comprises of an ensemble of leaveout-classifiers.”
Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectives (2022, methodology): “A number of methods (Liang et al., 2018; Vyas et al., 2018; Lee et al., 2018b) need OOD samples for tuning hyper-parameter selection, e.g., the threshold for verification.”
A survey of uncertainty in deep neural networks (2021, methodology): “2[31] 20[90] 24[234] 25[85] 26[86] 27[94] 28[39] 29[235] 30[40] 31[240] 32[13] 33[241] 34[246]”
Contrastive Training for Improved Out-of-Distribution Detection (2020, methodology): “MSP scoring was improved by using strategies like confidence loss [17], auxiliary objectives [6, 13, 23], margin loss [34] and outlier exposure [12].”
Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy (2019, methodology): “Following [16, 26], we implemented our network based on two state-of-the-art neural network architectures, DenseNet [10] and Wide ResNet (WRN) [32].”
Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “Enhancement [69], [112], [113], [114], [115], [116], [117], [118], [119], [120], [121], [122], [123], [124], [125]”
Semantically Coherent Out-of-Distribution Detection (2021, methodology): “Existing OOD detection methods mostly focus on calibrating the distribution of the softmax layer [4] through temperature scaling [7], generative models [8, 9], or ensemble methods [10, 11].”
Uncertainty Estimation for Language Reward Models (2022, background): “They have been found to be a strong baseline in a variety of applications [3, 47, 30, 15], although they have been criticized on both a theoretical [28] and empirical [29] basis.”
SSD: A Unified Framework for Self-Supervised Outlier Detection (2021, methodology): “Vyas et al. (2018) uses ensemble of leave-one-out classiﬁer, Winkens et al. (2020) uses contrastive self-supervised training, and Hendrycks et al. (2019b) uses rotation based self-supervised loss, in conjunction with supervised cross-entropy loss to achieve state-of-the-art performance in OOD…”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, methodology): “Ensembles of models have long been utilized to improve the performance in computer vision [20, 32, 34, 47, 50, 42].”
CODEs: Chamfer Out-of-Distribution Examples against Overconfidence Issue (2021, methodology): “Recent works improve OOD detection by using the ODIN score [31, 23], Mahalanobis distance [28], energy score [32], ensemble of multiple classifiers [42, 48], residual flow [51], generative models [38], self-supervised learning [20, 35] and gram matrices [39].”
A Less Biased Evaluation of Out-of-distribution Sample Detectors (2018, methodology): “Further extensions rely on statistics of hidden representations [25, 29], construct classiﬁer ensembles with subsets of data [37], or perform regression on word embeddings [34].”