Scaling Out-of-Distribution Detection for Real-World Settings

By Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Mohammadreza Mostajabi, Jacob Steinhardt, Dawn Song. 2022

690 citations recorded by the source corpus.

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as medical screening and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and hundreds of classes. To make future work in real-world settings possible, we also create a new benchmark for anomaly segmentation by introducing the Combined Anomalous Object Segmentation benchmark. Our novel benchmark combines two datasets for anomaly segmentation that incorporate both realism and anomaly diversity. Using both real images and those from a simulated driving environment, we ensure the background context and a wide variety of anomalous objects are naturally integrated, unlike before. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks we consider, establishing a new baseline for future work. These results, along with our new anomaly segmentation benchmark, open the door to future research in out-of-distribution detection.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Natural Adversarial Examples (2019, methodology): “Next we use the recent Maximum Logit detector [3].”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “Others have studied proprietary datasets with realistic distribution shifts, such as the StreetView StoreFronts dataset (Hendrycks et al., 2020) or diabetic retinopathy datasets (D’Amour et al., 2020).”
EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies (2023, background): “Visual anomaly detection is a rapidly growing area of research with a diverse range of applications, including medical imaging [3, 18, 36], autonomous driving [13, 23, 30], and industrial inspection [7, 17, 40].”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “To perform the detection, we use the maximum softmax probability (MSP) (Hendrycks et al., 2019; Tran et al., 2022).”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…et al., 2021; Baldock et al., 2021; Paul et al., 2021; Agarwal et al., 2021), prototypical or core subset selection (Paul et al., 2021; Sener and Savarese, 2018; Shim et al., 2021; Huggins et al., 2017; Sorscher et al., 2022) and out-of-distribution (Hendrycks et al., 2019; LeBrun et al., 2022).”
ViM: Out-Of-Distribution with Virtual-logit Matching (2022, methodology): “Scores are mainly derived from three sources: (1) the probability, such as the maximum softmax probabilities [13], the minimum KL-divergence between the softmax and the mean class-conditional distributions [12]; (2) the logit, such as the maximum logits [12], the logsumexp function over log-”
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection (2022, methodology): “To build OOD dataset, we use a 10k subset of Species [51] with 713k images, iNaturalist [52] with 10k images, ImageNet-O [53] with 2k images, and OpenImage-O [54] with 17k images.”
Generalized Out-of-Distribution Detection: A Survey (2021, positive): “PixMix achieves 93.1% on Near-OOD in CIFAR-10, the best performance among all the methods in this benchmark.”