Exploring the Limits of Out-of-Distribution Detection
By Stanislav Fort, Jie Jessie Ren, Balaji Lakshminarayanan. 2021
428 citations recorded by the source corpus.
Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-trained transformers can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% to 77% using transformers and unsupervised pre-training. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained transformers are particularly well-suited for outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained transformers such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information without any accompanying images, and show that this outperforms previous SOTA on standard vision OOD benchmark tasks.
Subfields: Artificial Intelligence, Anomaly & Outlier Detection
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Sergey Levine
- Trevor Darrell
- Neil Houlsby
In Anomaly & Outlier Detection
Citations in the corpus
- Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “20–30% shape bias / 70–80% texture bias) (Geirhos et al., 2) open-set recognition—also known as OOD detection (Fort et al., 2021), which we evaluate via the AUROC and AUPRC, with Places365 as the OOD dataset (Hendrycks et al., 2019); for more details, see Appendix I.”
- Generalized Out-of-Distribution Detection: A Survey (2021, positive): “Several concurrent works [140], [155], [156] demonstrate that strong pretrained transformers can significantly improve”
- Scaling Out-of-Distribution Detection for Real-World Settings (2022, methodology): “Recent work has suggested that stronger representations from Vision Transformers pre-trained on ImageNet-21K can make out-of-distribution detection trivial (Fort et al., 2021; Koner et al., 2021).”
- Is Out-of-Distribution Detection Learnable? (2022, background): “[45, 46] consider this issue and name it near-OOD detection.”
- A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection (2021, background): “It has been shown that the high-quality features learnt during this pre-training stage can be very useful in boosting the performance of the downstream task [7; 18; 5].”
- Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, background): “This attack is similar to the poisoning samples attacks, although they differ in that the poisoned training samples are not modifications of the original ones, but samples from outside the input distribution [56].”
- Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP (2021, methodology): “We are not aware of any existing zero-shot OOD detection model except (Fort, Ren, and Lakshminarayanan 2021) which requires unseen class labels to be given for detection (the paper’s main focus is not zero-shot OOD detection).”
- CMG: A Class-Mixed Generation Approach to Out-of-Distribution Detection (2022, background): “It has been shown recently that using pre-trained representations and few-shot outliers can improve the results [9].”
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “For this step, a separate BioViL-T is trained on ‘clean’ studies with single acquisitions and later used in a zero-shot setting to detect out-of-distribution samples [26,27] arising from the reimaging process.”
- Adversarial vulnerability of powerful near out-of-distribution detection (2022, methodology): “[13] introduce a new kind of OOD detection scenario, where they use a multi-modal CLIP model [16].”
- WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation (2023, background): “Additionally, Vision Transformer (ViT) have recently shown its potential in visual inspection [9, 25].”