Do Adversarially Robust ImageNet Models Transfer Better?

By Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, Aleksander Madry. Neural Information Processing Systems, 2020

492 citations recorded by the source corpus.

Transfer learning is a widely-used paradigm in deep learning, where models pre-trained on standard datasets can be efficiently adapted to downstream tasks. Typically, better pre-trained models yield better transfer results, suggesting that initial accuracy is a key aspect of transfer learning performance. In this work, we identify another such aspect: we find that adversarially robust models, while less accurate, often perform better than their standard-trained counterparts when used for transfer learning. Specifically, we focus on adversarially robust ImageNet classifiers, and show that they yield improved accuracy on a standard suite of downstream classification tasks. Further analysis uncovers more differences between robust and standard models in the context of transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations. Our code and models are available at this https URL .

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/17293cd36ee5e7ec37dcec1d5ab85f9b77ad65d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

How Well Do Sparse ImageNet Models Transfer? (2021, methodology): “[61] considered whether adversarially robust ImageNet classifiers can outperform standard ones for transfer learning, and find that this can indeed be the case.”
Partial success in closing the gap between human and machine vision (2021, methodology): “In order to disentangle the influence of objective function, architecture and training dataset size, we tested a total of 52 models: 24 standard ImageNet-trained CNNs [37], 8 selfsupervised models [38–43],2 6 Big Transfer models [45], 5 adversarially trained models [46], 5 vision transformers [47, 4”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “[286] that adversarially trained models, while less accurate than the standard models, often perform better for transfer learning.”
On the Effectiveness of Parameter-Efficient Fine-Tuning (2022, background): “Moreover, many research works (Salman et al., 2020; Jiang et al., 2020; Li and Zhang, 2021; Hua et al., 2021) propose to train model adversarially while some researchers (You et al., 2019; Liang et al., 2021; Ansell et al., 2021; Chen et al., 2021) propose to utilize the lottery ticket approaches…”
Frustratingly Easy Transferability Estimation (2021, unknown): “The source datasets are ImageNet [32] and 10 image datasets from [33], including Caltech-101, Caltech-256, DTD, Flowers, SUN397, Pets, Food, Aircraft, Birds and Cars.”
Concept Generalization in Visual Representation Learning (2020, unknown): “and r-ReLabel [70]) and adversarial robustness (r-AdvRobust [48]) models.”
Does language help generalization in vision models? (2021, background): “Moreover, [23] showed that these robust features also improve transfer learning; here we find that these benefits extend to the few-shot and unsupervised settings.”
Improved OOD Generalization via Adversarial Training and Pre-training (2021, methodology): “Following (Salman et al., 2020a), we pre-train the model on ImageNet and then ﬁne-tune it on CIFAR10 .”
Overcoming Negative Transfer: A Survey (2020, background): “…latent space of features form source and target domains to transfer common knowldege, such as part-based ensemble [40]; (iv) Enhancing the robustness of transfer learning process, such as noise simulation [91], pseudo label noise robustness [90], [92], and adversarial sample robustness [83].”
Diffusion Models for Adversarial Purification (2022, unknown): “(Engstrom et al., 2019) 29.22 31.06 (Wong et al., 2020) 26.24 26.95 (Salman et al., 2020) 34.96 37.89 WideResNet-50-2 (Salmanetal.,2020) 38.14 39.25 C.3.”
Geometry-aware Instance-reweighted Adversarial Training (2020, background): “Furthermore, there is still a large room to apply adversarial training techniques into other domains such as pre-training (Hendrycks et al., 2019; Chen et al., 2020; Jiang et al., 2020; Salman et al., 2020), noisy labels (Zhu et al., 2021) and so on.”