Understanding Robustness of Transformers for Image Classification

By Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, Andreas Veit. 2021

515 citations recorded by the source corpus.

Deep Convolutional Neural Networks (CNNs) have long been the architecture of choice for computer vision tasks. Recently, Transformer-based architectures like Vision Transformer (ViT) have matched or even surpassed ResNets for image classification. However, details of the Transformer architecture –such as the use of non-overlapping patches– lead one to wonder whether these networks are as robust. In this paper, we perform an extensive study of a variety of different measures of robustness of ViT models and compare the findings to ResNet baselines. We investigate robustness to input perturbations as well as robustness to model perturbations. We find that when pre-trained with a sufficient amount of data, ViT models are at least as robust as the ResNet counterparts on a broad range of perturbations. We also find that Transformers are robust to the removal of almost any single layer, and that while activations from later layers are highly correlated with each other, they nevertheless play an important role in classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “Follow up works [2, 29] extend the robustness study on ViTs to much common image corruption and distribution shift, and indicate ViTs are more robust learners.”
Are Transformers More Robust Than CNNs? (2021, methodology): “Though both [4] and [35] claim that Transformers are preferable to CNNs in terms of robustness, we ﬁnd that such conclusion cannot be strongly drawn based on their existing experiments.”
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers (2021, background): “Some recent works [73], [74] have found that the vision transformer [61], [66], [67] demonstrates stronger performance and better robustness to input disturbances than CNNs [16], [17].”
DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation (2021, background): “This might be due to the shape-bias of Transformers [2], which causes the network to focus more on shape than texture.”
Vision Transformers are Robust Learners (2021, background): “Various recent works have opened up the investigation on evaluating the robustness of ViTs (Bhojanapalli et al. 2021; Shao et al. 2021; Mahmood, Mahmood, and Van Dijk 2021) but with a relatively limited scope.”
Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers (2021, methodology): “Bhojanapalli et al. (2021) investigate several different ViT and ResNet models to understand the robustness of the ViT models.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “Furthermore, the robustness study [51] of ViTs is extended to the natural distribution shift and common image corruption, which demonstrate the superiority of ViTs over CNNs in the robustness benchmark.”
On Improving Adversarial Transferability of Vision Transformers (2021, background): “In this work, we investigate whether the weak transferability of adversarial patterns from high-performing ViT models, as reported in recent works [13, 14, 15], is a result of weak features or a weak attack.”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Since the two modules extract features in a complementary manner, it is interesting to explore whether such design will make the backbone network robust to adversarial attack (Bhojanapalli et al., 2021).”
Intriguing Properties of Vision Transformers (2021, background): “[16] study ViT against spatial perturbations [15] and its robustness to removal of any single layer.”
How Do Vision Transformers Work? (2022, background): “Figure 10b shows the results of lesion study (Bhojanapalli et al., 2021), where one NN unit is removed from already trained ResNet and Swin during the testing phase.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, background): “Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7].”
Blending Anti-Aliasing into Vision Transformer (2021, background): “In [52], people ﬁnd that the CLS token in ViT [1] changes slowly at earlier layers, but evolves rapidly in later layers where limited updates happen to representations of individual patches.”