Vision Transformers are Robust Learners

By Sayak Paul, Pin-Yu Chen. 2021

386 citations recorded by the source corpus.

Transformers, composed of multiple self-attention layers, hold strong promises toward a generic learning primitive applicable to different data modalities, including the recent breakthroughs in computer vision achieving state-of-the-art (SOTA) standard accuracy. What remains largely unexplored is their robustness evaluation and attribution. In this work, we study the robustness of the Vision Transformer (ViT) (Dosovitskiy et al. 2021) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We use six different diverse ImageNet datasets concerning robust classification to conduct a comprehensive performance comparison of ViT(Dosovitskiy et al. 2021) models and SOTA convolutional neural networks (CNNs), Big-Transfer (Kolesnikov et al. 2020). Through a series of six systematically designed experiments, we then present analyses that provide both quantitative andqualitative indications to explain why ViTs are indeed more robust learners. For example, with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1accuracy of 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT. Our analyses on image masking, Fourier spectrum sensitivity, and spread on discrete cosine energy spectrum reveal intriguing properties of ViT attributing to improved robustness. Code for reproducing our experiments is available at https://git.io/J3VO0.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “Follow up works [2, 29] extend the robustness study on ViTs to much common image corruption and distribution shift, and indicate ViTs are more robust learners.”
Simpler is Better: off-the-shelf Continual Learning Through Pretrained Backbones (2022, background): “This also follows the conclusions made by Paul and Chen [20] on the fact that vision transformers are more robust learners with respect to CNNs.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “They provided empirical results to demonstrate that, MHSA is more flexible [50], capable (less inductive bias) [20], more robust to distortions [66, 98], or able to model long-range dependencies [69, 90].”
DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation (2021, background): “Based on recent findings [3, 51, 53] and an architecture comparison for UDA, which we will present in Sec.”
Exploring the Limits of Out-of-Distribution Detection (2021, background): “[2021] show that ViT is more robust to distribution shift and natural adversarial examples [Paul and Chen, 2021], and Mao et al.”
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts (2021, background): “Despite the encouraging performance achieved on standard benchmarks and several properties revealed in recent works [1, 5, 23, 24], the generalization ability of Vision Transformers (ViTs) is still less understood.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, background): “Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7].”
Multimodal Learning With Transformers: A Survey (2022, background): “Although that recent attempts [99], [182], [289], [290] study and evaluate how the Transformer components/sub-layers con-”
Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization (2022, result): “The conclusion remains in line with previous works [35].”
A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection (2021, background): “It has been shown that the high-quality features learnt during this pre-training stage can be very useful in boosting the performance of the downstream task [7; 18; 5].”
Are Transformers More Robust Than CNNs? (2021, result): “In addition, we note this conclusion is further corroborated in concurrent works [58, 30, 50, 62, 29], where a range of additional out-of-distribution tasks/datasets are tested.”
StyTr2: Image Style Transfer with Transformers (2021, background): “Second, the transformer architecture models relationships in input shapes [41], and different layers extract similar structural information [46] (see Fig.”
How Do Vision Transformers Work? (2022, background): “2 ViTs are robust against data corruptions, image occlusions (Naseer et al., 2021), and adversarial attacks (Shao et al., 2021; Bhojanapalli et al., 2021; Paul & Chen, 2022; Mao et al., 2021).”
Artificial intelligence in histopathology: enhancing cancer research and clinical oncology (2022, background): “Shortly after their emergence, trans-formers outperformed CNNs in many computer vision tasks and proved more robust against adversarial attacks 120,121 .”
Intriguing Properties of Vision Transformers (2021, background): “A concurrent work from Paul and Chen [18] also develops similar insights on robustness of ViTs but with a somewhat different set of experiments.”