Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

By Dan Hendrycks, Thomas G. Dietterich. 2019

4,555 citations recorded by the source corpus.

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “They distinguish these datasets, which all consist of novel images collected from a variety of sources, from synthetic distribution shifts such as ImageNet-C (Hendrycks & Dietterich, 2019), Stylized ImageNet (Geirhos et al., 2018), or adversarial attacks (Goodfellow et al., 2014) which are created b”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “and past work show that increases in accuracy do not directly translate to reduced IID/OOD generalization gaps (Hendrycks and Dietterich, 2019; Fried et al., 2019).”
Shortcut learning in deep neural networks (2020, background): “While adversarial attacks test model performance on model-dependent worst-case noise, a related line of research focuses on model-independent noise like image corruptions [51, 52].”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “[68, 24, 55, 22], have shown that computer vision models lack robustness.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, methodology): “We show results from the evaluation on several ImageNet-C tasks in Figure 4.”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “In Figure 5, we analyze the predictiveness of one commonly studied synthetic robustness metric: average accuracy on image corruptions [38].”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, methodology): “Recent works [12, 26, 14] have begun to characterize model performance on out-ofdistribution (OOD) data with various new test sets, with dissonant findings.”
Do Image Classifiers Generalize Across Time? (2019, background): “One model for these corruptions are the perturbations introduced in [13].”
Learning Robust Global Representations by Penalizing Local Predictive Power (2019, methodology): “Also, notably, different from perturbed ImageNet validation sets (Geirhos et al., 2019; Hendrycks and Dietterich, 2019), the images of ImageNet-Sketch are collected independently from the original validation images.”
Do ImageNet Classifiers Generalize to ImageNet? (2019, neutral): “[23] Dan Hendrycks and Thomas Dietterich.”
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness (2018, methodology): “Furthermore, we provide robustness results for our models tested on ImageNet-C, a comprehensive benchmark of 15 different image corruptions (Hendrycks & Dietterich, 2019), in Table 4 of the Appendix.”
Towards Robust Vision Transformer (2021, methodology): “2) for common corruption robustness, we adopt ImageNet-C [17] which consists of 15 types of algorithmically generated corruptions with five levels of severity.”
On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation (2021, background): “Despite these advances, generative methods continue to be detectable [10, 74], and discriminative networks continue to be sensitive to small perturbations, such as shifts [3, 22] and JPEG compression [29].”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We measure the top-1 accuracy except for ImageNet-C.”
Aligning AI With Shared Human Values (2020, background): “Future work should also make sure these models are explainable, and should test model robustness to adversarial examples and distribution shift (Goodfellow et al., 2014; Hendrycks and Dietterich, 2019).”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, methodology): “Hendrycks & Dietterich (2019) show that classi-ﬁcation error of modern models rises from 22% on the usual ImageNet test set to 64% on ImageNet-C, a test set consisting of various corruptions applied to ImageNet test images.”
Unsolved Problems in ML Safety (2021, background): “In addition to existing robustness benchmarks [78, 102, 75], researchers could create more environments and benchmarks to stress-test systems, ﬁnd their breaking points, and determine whether they will function appropriately in potential future scenarios.”