Intriguing Properties of Vision Transformers

By Muzammal Naseer, Kanchana Ranasinghe, Salman Hameed Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang. 2021

825 citations recorded by the source corpus.

Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. These models are based on multi-head self-attention mechanisms that can flexibly attend to a sequence of image patches to encode contextual cues. An important question is how such flexibility in attending image-wide context conditioned on a given patch can facilitate handling nuisances in natural images e.g., severe occlusions, domain shifts, spatial permutations, adversarial and natural perturbations. We systematically study this question via an extensive set of experiments encompassing three ViT families and comparisons with a high-performing convolutional neural network (CNN). We show and analyze the following intriguing properties of ViT: (a) Transformers are highly robust to severe occlusions, perturbations and domain shifts, e.g., retain as high as 60% top-1 accuracy on ImageNet even after randomly occluding 80% of the image content. (b) The robust performance to occlusions is not due to a bias towards local textures, and ViTs are significantly less biased towards textures compared to CNNs. When properly trained to encode shape-based features, ViTs demonstrate shape recognition capability comparable to that of human visual system, previously unmatched in the literature. (c) Using ViTs to encode shape representation leads to an interesting consequence of accurate semantic segmentation without pixel-level supervision. (d) Off-the-shelf features from a single ViT model can be combined to create a feature ensemble, leading to high accuracy rates across a range of classification datasets in both traditional and few-shot learning paradigms. We show effective features of ViTs are due to flexible and dynamic receptive fields possible via the self-attention mechanism.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “[145] investigate multiple intriguing properties of ViTs in terms of their generalization and robustness.”
Transformers in Remote Sensing: A Survey (2022, background): “Recent works [2], [3] have shown that ViTs possess content-dependent long-range interaction modeling capabilities and can flexibly adjust their receptive fields to counter nuisances in data and learn effective feature representations.”
MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer (2023, background): “With the advancement of deep learning techniques, several studies have successfully applied neural network-based models, including classical convolutional neural networks (CNNs) (Ji et al. 2021; Wu et al. 2022b) and the recently popular vision transformers (ViTs)(Chen et al. 2021; Wang et al. 2021b)”
Delving Deep into the Generalization of Vision Transformers under Distribution Shifts (2021, background): “Despite the encouraging performance achieved on standard benchmarks and several properties revealed in recent works [1, 5, 23, 24], the generalization ability of Vision Transformers (ViTs) is still less understood.”
Wavelet Convolutions for Large Receptive Fields (2024, result): “This is unlike attention heads, which are known to be more attuned to low frequencies, as demonstrated in other studies [42,43,54].”
Partial success in closing the gap between human and machine vision (2021, result): “In line with [72, 73], we observe a higher shape bias compared to most standard CNNs.”
Exploring the Limits of Out-of-Distribution Detection (2021, background): “Caron et al. [2021] demonstrate many emerging properties in self-supervised ViTs, while Shao et al. [2021], Mahmood et al. [2021] show that they are more robust to adversarial perturbations, and Naseer et al. [2021] that they have less texture bias.”
How Do Vision Transformers Work? (2022, background): “The results thus suggests that MSAs are shape-biased (Naseer et al., 2021), whereas Convs are texture-biased (Geirhos et al.”
Efficient Training of Visual Transformers with Small Datasets (2021, background): “[10] show that the (absolute) positional embedding used in ViT [4] is not necessary for the transformer to solve very challenging occlusion or patch permutation tasks, and they conclude that these tasks are solved by ViT thank to its “dynamic receptive field” (i.”
Transformers in Medical Imaging: A Survey (2022, background): “Furthermore, recent research indicate that the prediction errors of ViTs are more consistent with those of humans than CNNs [29]–[32].”
OW-DETR: Open-world Detection Transformer (2021, background): “The advent of transformers for natural language processing [33, 35] has inspired studies to investigate related ideas for vision tasks [8,10,16,24], including standard object detection [3,37].”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, background): “Naseer et al. (2021) also showed that ViT are robust to patch drop, which motivates the idea of dropping the less informative patches to accelerate ViT inference.”
iBOT: Image BERT Pre-Training with Online Tokenizer (2021, methodology): “For occlusion, we study the linear accuracy with salient and non-salient patch dropping following Naseer et al. (2021) with an information loss ratio of 0.5. iBOT has a smaller performance drop under both settings.”
Efficient Training of Audio Transformers with Patchout (2021, background): “In computer vision, the idea of removing patches during inference was investigated to assess the vision transformer’s robustness against input perturbations [15].”
DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation (2021, background): “Based on recent findings [3, 51, 53] and an architecture comparison for UDA, which we will present in Sec.”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “It is because the Transformer is weak in shape modeling caused by its hard patch partition (Naseer et al. 2021).”
Are Transformers More Robust Than CNNs? (2021, result): “In addition, we note this conclusion is further corroborated in concurrent works [58, 30, 50, 62, 29], where a range of additional out-of-distribution tasks/datasets are tested.”