Towards Robust Vision Transformer

By Xiaofeng Mao, Gege Qi, YueFeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, Hui Xue. 2021

245 citations recorded by the source corpus.

Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and computation cost, lacking the investigation of the intrinsic influence on model robustness and generalization. In this work, we conduct systematic evaluation on components of ViTs in terms of their impact on robustness to adversarial examples, common corruptions and distribution shifts. We find some components can be harmful to robustness. By leveraging robust components as building blocks of ViTs, we propose Robust Vision Transformer (RVT), which is a new vision transformer and has superior performance with strong robustness. Inspired by the findings during the evaluation, we further propose two new plug-and-play techniques called position-aware attention scaling and patch-wise augmentation to augment our RVT, which we abbreviate as RVT*. The experimental results of RVT on ImageNet and six robustness benchmarks demonstrate its advanced robustness and generalization ability compared with previous ViTs and state-of-the-art CNNs. Furthermore, RVT-S* achieves Top-1 rank on multiple robustness leaderboards including ImageNet-C, ImageNet-Sketch and ImageNet-R.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Safety, Robustness & Interpretability

Citations in the corpus

A ConvNet for the 2020s (2022, background): “ConvNeXt (in particular the large-scale model variants) exhibits promising robustness behaviors, outperforming state-of-the-art robust transformer models [44] on several benchmarks.”
FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “Performance of competing models is reported by [38] and [36].”
Exploring the Limits of Out-of-Distribution Detection (2021, background): “Paul and Chen [2021], Minderer et al. [2021] show that ViT is more robust to distribution shift and natural adversarial examples [Paul and Chen, 2021], and Mao et al. [2021] propose a robust ViT.”
Rep ViT: Revisiting Mobile CNN From ViT Perspective (2023, methodology): “Previous works [5, 57] optimize the inference speed of models based on metrics like floating point operations (FLOPs) or model sizes.”
MEMO: Test Time Robustness via Adaptation and Augmentation (2021, methodology): “The parameters for the small robust vision transformer (RVT∗-small) model are obtained from https://drive.google.com/file/d/1g40huqDVthjS2H5sQV3ppcfcWEzn9ekv.”
MobileOne: An Improved One millisecond Mobile Backbone (2022, methodology): “We follow the protocol set by [44] for all the evaluations.”
How Do Vision Transformers Work? (2022, background): “It also provides an explanation of how adding Convs to Transformer’s MLP block improves accuracy and robustness (Yuan et al., 2021; Guo et al., 2021; Mao et al., 2021).”
Multimodal Learning With Transformers: A Survey (2022, result): “Although that recent attempts [99], [182], [289], [290] study and evaluate how the Transformer components/sub-layers con-”