Contrastive Training for Improved Out-of-Distribution Detection
By Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, J. Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon A. A. Kohl and 3 more. 2020
271 citations recorded by the source corpus.
Reliable detection of out-of-distribution (OOD) inputs is increasingly understood to be a precondition for deployment of machine learning systems. This paper proposes and investigates the use of contrastive training to boost OOD detection performance. Unlike leading methods for OOD detection, our approach does not require access to examples labeled explicitly as OOD, which can be difficult to collect in practice. We show in extensive experiments that contrastive training significantly helps OOD detection performance on a number of common benchmarks. By introducing and employing the Confusion Log Probability (CLP) score, which quantifies the difficulty of the OOD detection task by capturing the similarity of inlier and outlier datasets, we show that our method especially improves performance in the `near OOD' classes -- a particularly challenging setting for previous methods.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Anomaly & Outlier Detection
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Bing Liu
- Yixuan Li
- Wojciech Samek
- Yoshua Bengio
- Sergey Levine
- Stanislav Fort
- Grgoire Montavon
- Mohit Bansal
In Representation & Self-Supervised Learning
- Bing Liu
- Yixuan Li
- Sepideh Esmaeilpour
- Wojciech Samek
- Lei Shu
- Grgoire Montavon
- Jinwoo Shin
- Negar Rostamzadeh
In Anomaly & Outlier Detection
- Stanislav Fort
- Yixuan Li
- Rui Huang
- Mung Chiang
- Sepideh Esmaeilpour
- Prateek Mittal
- Thomas G. Dietterich
- Lei Shu
Citations in the corpus
- A Unifying Review of Deep and Shallow Anomaly Detection (2020, background): “Other recent and promising self-supervised approaches are based on contrastive learning [156], [545], [550].”
- CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, background): “[68] and Liu and Abbeel [40] report that contrastive learning also improves the OOD detection performance of classifiers [39, 38, 25].”
- Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP (2021, background): “Recently, some authors made a distinction between hard and easy OOD detection problems (Winkens et al. 2020).”
- A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection (2021, background): “…in identifying far OOD samples (samples which are semantically and stylistically very different from the in-distribution samples, e.g., CIFAR-10 vs. SVHN), we identify that it often fails for near OOD samples (samples which are semantically similar to the in-distribution samples [21], e.g.,”
- On the Importance of Gradients for Detecting Distributional Shifts in the Wild (2021, background): “A plethora of literature has emerged to utilize generative models for OOD detection [19, 37, 39, 40, 45, 46, 48].”
- SSD: A Unified Framework for Self-Supervised Outlier Detection (2021, methodology): “Most of these approaches model features of in-distribution data at output [30, 19, 10] or in the feature space [29, 47] for detection.”
- Evaluation Gaps in Machine Learning Practice (2022, methodology): “Our framework for analyzing the gaps builds upon and complements other streams of work on ML evaluation practices, including addressing distribution shifts between development data and application data [34, 94, 160], and robustness to perturbations in test items [118, 132, 173 ?”
- Exploring the Limits of Out-of-Distribution Detection (2021, methodology): “…focus on the difficult near-OOD as this is a more challenging and realistic problem; many methods can achieve high AUROC on the easier far-OOD benchmarks, but do not perform as well in near-OOD tasks, cf. [Winkens et al., 2020, Table 1] which compares many methods on near-OOD and far-OOD tasks.”
- Hybrid Discriminative-Generative Training via Contrastive Learning (2020, methodology): “Comparing with contrastive learning approach (Winkens et al., 2020), HDGE differs in that the contrastive loss inside the log q(x|y) utilizes label information to help contrast similar data points.”