A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

By Kimin Lee, Kibok Lee, Honglak Lee, Jinwoo Shin. 2018

2,720 citations recorded by the source corpus.

Detecting test samples drawn sufficiently far away from the training distribution statistically or adversarially is a fundamental requirement for deploying a good classifier in many real-world machine learning applications. However, deep neural networks with the softmax classifier are known to produce highly overconfident posterior distributions even for such abnormal samples. In this paper, we propose a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier. We obtain the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance. While most prior methods have been evaluated for detecting either out-of-distribution or adversarial samples, but not both, the proposed method achieves the state-of-the-art performances for both cases in our experiments. Moreover, we found that our proposed method is more robust in harsh cases, e.g., when the training dataset has noisy labels or small number of samples. Finally, we show that the proposed method enjoys broader usage by applying it to class-incremental learning: whenever out-of-distribution samples are detected, our classification rule can incorporate new classes well without further training deep models.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, background): “Recently, a number of reserchs are proposed to enhance the performance of the baseline detector [6, 5] but in this paper, we follow only the baseline detector algorithm without any input enhancement and temperature scaling [6].”
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning (2020, background): “A natural question then arises: how can we construct a dataset satisfying such a condition? Inspired by [73], we propose the following algorithm.”
Using Pre-Training Can Improve Model Robustness and Uncertainty (2019, methodology): “Lee et al. (2018b) propose Mahalanobis distance-based scores which characterize out-of-distribution samples using hidden features.”
A Theoretical Study on Solving Continual Learning (2022, methodology): “ODIN : Researchers have proposed several methods to improve the OOD detection performance of a trained network by post-processing [7, 70, 71].”
Few-Shot Continual Active Learning by a Robot (2022, background): “Another related field to active learning is open-set recognition (or out of the distribution detection (OOD)) [37, 38, 38, 39], in which a model that is trained on a set of tasks/classes might face samples belonging previously unknown classes in the test set.”
Adaptive Orthogonal Projection for Batch and Online Continual Learning (2022, methodology): “In testing, a distance function over the nearest exemplar/class mean is used for classiﬁcation (Rebufﬁ et al. 2017; Lee et al. 2018; Javed and Shafait 2018; Bendale and Boult 2015).”
Continual Learning by Using Information of Each Class Holistically (2021, methodology): “To be more complete, we also compare with two traditional methods of class incremental learning based on nearest-mean: iCaRL (Rebufﬁ, Kolesnikov, and Lampert 2017) and the system in (Lee et al. 2018).”
Self-Supervised Training Enhances Online Continual Learning (2021, background): “We showed that self-supervised features generalize to both unseen classes and unseen datasets in online settings, so extending our work to open world learning would only require the addition of a component to identify if samples are unknown or out-of-distribution [41, 31, 36].”
Distilling Causal Effect of Data in Class-Incremental Learning (2021, background): “For example, if the new data are outof-distribution compared to the old one, the features of the new data extracted from the old network will be elusive, and thus the corresponding distillation loss will mislead the new training [23, 20].”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, background): “An alternative, competitive approach is Mahalanobis Distance (Lee et al., 2018), which computes a distance in the feature space of a pretrained softmax/cross entropy model in combination with a number of dataset speciﬁc augmentations that rely on tuning via the out of distribution dataset or in…”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Methods with an OOD-detection component in addition to p ( y | x ) (Bishop, 1994; Lee et al., 2018; Liang et al., 2018), and related work on selective classiﬁcation (Geifman & El-Yaniv, 2017).”
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis (2019, methodology): “Further, it was shown in [33] that the estimated posterior distribution of LDA is equivalent to the softmax classiﬁer often used with modern neural networks, thus motivating the use of SLDA.”
Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer (2021, positive): “Therefore, we include Out-of-Distribution (OoD) Detection [11, 12, 13] to select unlabeled data corresponding to the classes our learner has seen so far with high conﬁdence.”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, methodology): “They propose mechanism to calibrate output conﬁdences in deep models (Lee et al., 2018a), formulate a discrepancy loss between knowns and known unknowns (Yu and Aizawa, 2019) or modify the embedding to explicitly separate them.”