Robustness properties of Facebook's ResNeXt WSL models

By Emin Orhan. 2019

35 citations recorded by the source corpus.

We investigate the robustness properties of ResNeXt class image recognition models trained with billion scale weakly supervised data (ResNeXt WSL models). These models, recently made public by Facebook AI, were trained with ~1B images from Instagram and fine-tuned on ImageNet. We show that these models display an unprecedented degree of robustness against common image corruptions and perturbations, as measured by the ImageNet-C and ImageNet-P benchmarks. They also achieve substantially improved accuracies on the recently introduced "natural adversarial examples" benchmark (ImageNet-A). The largest of the released models, in particular, achieves state-of-the-art results on ImageNet-C, ImageNet-P, and ImageNet-A by a large margin. The gains on ImageNet-C, ImageNet-P, and ImageNet-A far outpace the gains on ImageNet validation accuracy, suggesting the former as more useful benchmarks to measure further progress in image recognition. Remarkably, the ResNeXt WSL models even achieve a limited degree of adversarial robustness against state-of-the-art white-box attacks (10-step PGD attacks). However, in contrast to adversarially trained models, the robustness of the ResNeXt WSL models rapidly declines with the number of PGD steps, suggesting that these models do not achieve genuine adversarial robustness. Visualization of the learned features also confirms this conclusion. Finally, we show that although the ResNeXt WSL models are more shape-biased than comparable ImageNet-trained models in a shape-texture cue conflict experiment, they still remain much more texture-biased than humans, suggesting that they share some of the underlying characteristics of ImageNet-trained models that make this benchmark challenging.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Similarly, Transformer models are pretrained with large amounts of data, which may also aid robustness (Orhan, 2019; Xie et al., 2020; Hendrycks et al., 2019a).”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, background): “Existing work typically argues that an intervention improves robustness by showing that the accuracy on a robustness test set has improved (e.g., see [34, 40, 63, 102, 115]).”
MEMO: Test Time Robustness via Adaptation and Augmentation (2021, methodology): “Largely separate from these frameworks, various empirical methods have also been proposed for dealing with shift, such as increasing the model and training dataset size or using heavy training augmentations [28, 40, 12].”
Towards Stable Test-Time Adaptation in Dynamic Wild World (2023, methodology): “…at training time , including domain generalization (Shankar et al., 2018; Li et al., 2018a; Dou et al., 2019), increasing the training dataset size (Orhan, 2019), various data augmentation techniques (Lim et al., 2019; Hendrycks et al., 2020; Li et al., 2021; Hendrycks et al., 2021; Yao et al.,…”