Natural Adversarial Examples

By Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, Dawn Song. 2019

1,984 citations recorded by the source corpus.

We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.

Subfields: Artificial Intelligence, Hardware and Architecture, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, background): “They measure performance on a set of 7 distribution shifts: ImageNetV2 (Recht et al., 2019), ImageNet Sketch (Wang et al., 2019), Youtube-BB and ImageNet-Vid (Shankar et al., 2019), ObjectNet (Barbu et al., 2019), ImageNet Adversarial (Hendrycks et al., 2019), and ImageNet Rendition (Hendrycks et al”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Similar to computer vision (Orhan, 2019; Xie et al., 2020; Hendrycks et al., 2019a), pretraining on larger and more diverse datasets can improve robustness.”
Measuring Massive Multitask Language Understanding (2020, background): “Another calibration measure is the Root Mean Squared (RMS) calibration error (Hendrycks et al., 2019a; Kumar et al., 2019).”
On Robustness and Transferability of Convolutional Neural Networks (2020, methodology): “A [28] 7500 200 I MAGE N ET -”
Self-Training With Noisy Student Improves ImageNet Classification (2019, methodology): “The biggest gain is observed on ImageNet-A: our method achieves 3.5x higher accuracy on ImageNet-A, going from 16.6% of the previous state-of-the-art to 74.2% top-1 accuracy.”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “vision literatures where examples are stratified by topic or notions of difficulty (Hendrycks et al., 2019; Zellers et al., 2018).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “ImageNet-A [39] was created by classifying a set of images with a ResNet50 and only keeping the misclassiﬁed images.”
Towards Robust Vision Transformer (2021, methodology): “ImageNet-A [19] is used for evaluating the model under natural adversarial example.”
Predictability and Surprise in Large Generative Models (2022, background): “This should take the form of both static benchmarks (for example, adversarial datasets to probe for weaknesses in computer vision systems [34]), as well as continuous evaluation by humans carrying out multi-step interactions (e.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, methodology): “Accuracy is averaged over ImageNet and the ﬁve distribution shifts described in Section 3.1. the ﬁve natural distribution shifts: ImageNetV2 (Recht et al., 2019), ImageNet-R (Hendrycks et al., 2021a), ImageNet-Sketch (Wang et al., 2019), ObjectNet (Barbu et al., 2019), and ImageNet-A (Hendrycks et a”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Robustness evaluations on benchmarks with natural distribution shifts (Hendrycks et al., 2021b,a; Recht et al., 2019; Barbu et al., 2019; Wang et al., 2019) show that the accuracy of models like CLIP and ALIGN typically drops less than 10%, while the accuracy of supervised and semi-supervised…”
LiT: Zero-Shot Transfer with Locked-image text Tuning (2021, methodology): “We evaluate robustness on ImageNetv2 [48], -R [26, 63], -A [27], -ReaL [2], and ObjectNet [1], following CLIP and ALIGN.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (2021, background): “…ILSVRC-2012 benchmark (Deng et al., 2009) and its variants including ImageNet-R(endition) (Hendrycks et al., 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al., 2021) (more challenging images for ML models), and ImageNet-V2 (Recht et al., 2019).”
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement (2023, methodology): “We also evaluate ImageNet models on other distribution shift datasets, ImageNet-A [24], ImageNet-R [22], ImageNet-Sketch [61], ObjectNet [4], and ImageNet-C [23].”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, methodology): “We follow [94] and evaluate robustness performance on ImageNet distribution shift datasets [3, 23, 25, 61, 82].”
Unsolved Problems in ML Safety (2021, background): “[85] Dan Hendrycks, Kevin Zhao, Steven Basart, J.”