On the Robustness of Vision Transformers to Adversarial Examples

By Kaleel Mahmood, Rigel Mahmood, Marten van Dijk. 2021

270 citations recorded by the source corpus.

Recent advances in attention-based networks have shown that Vision Transformers can achieve state-of-the-art or near state-of-the-art results on many image classification tasks. This puts transformers in the unique position of being a promising alternative to traditional convolutional neural networks (CNNs). While CNNs have been carefully studied with respect to adversarial attacks, the same cannot be said of Vision Transformers. In this paper, we study the robustness of Vision Transformers to adversarial examples. Our analyses of transformer security is divided into three parts. First, we test the transformer under standard white-box and black-box attacks. Second, we study the transfer-ability of adversarial examples between CNNs and trans-formers. We show that adversarial examples do not readily transfer between CNNs and transformers. Based on this finding, we analyze the security of a simple ensemble defense of CNNs and transformers. By creating a new attack, the self-attention blended gradient attack, we show that such an ensemble is not secure under a white-box adversary. However, under a black-box adversary, we show that an ensemble can achieve unprecedented robustness without sacrificing clean accuracy. Our analysis for this work is done using six types of white-box attacks and two types of black-box attacks. Our study encompasses multiple Vision Transformers, Big Transfer Models and CNN architectures trained on CIFAR-10, CIFAR-100 and ImageNet.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yunhe Wang
2. Kai Han
3. Chunjing Xu
4. Jianyuan Guo
5. Dacheng Tao
6. Yehui Tang
7. Yuan He
8. Gege Qi

In Computer Vision and Pattern Recognition

1. Yunhe Wang
2. Kai Han
3. Jianyuan Guo
4. Chunjing Xu
5. Yehui Tang
6. Dacheng Tao
7. Zhenhua Liu
8. Xinghao Chen

In Molecular Biology

1. Dacheng Tao
2. Pin-Yu Chen
3. Chunjing Xu
4. Yunhe Wang

In Safety, Robustness & Interpretability

1. Dacheng Tao
2. Pin-Yu Chen
3. Yunhe Wang
4. Sayak Paul
5. Huan Zhang
6. Cho-Jui Hsieh

Citations in the corpus

• Towards Robust Vision Transformer (2021, background): “They find that ViTs are more adversarially robust than CNNs [34] and the transferability of adversarial examples between CNNs and ViTs is remarkably low [27].”
• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “Additionally, the adversarial transferability between CNNs and Transformers is unexpectedly low[50].”
• A Survey on Vision Transformer (2020, background): “Although the robustness has been investigated in [238], [239], [240], it is still an open problem waiting to be solved.”
• Vision Transformers are Robust Learners (2021, background): “[12] studied adversarial robustness of ViTs through various white-box, black-box and transfer attacks and found that model ensembling can achieve unprecedented robustness without sacrificing clean accuracy against a black-box adversary.”