How Do Vision Transformers Work?

By Namuk Park, Songkuk Kim. International Conference on Learning Representations, 2022

695 citations recorded by the source corpus.

The success of multi-head self-attentions (MSAs) for computer vision is now indisputable. However, little is known about how MSAs work. We present fundamental explanations to help better understand the nature of MSAs. In particular, we demonstrate the following properties of MSAs and Vision Transformers (ViTs): (1) MSAs improve not only accuracy but also generalization by flattening the loss landscapes. Such improvement is primarily attributable to their data specificity, not long-range dependency. On the other hand, ViTs suffer from non-convex losses. Large datasets and loss landscape smoothing methods alleviate this problem; (2) MSAs and Convs exhibit opposite behaviors. For example, MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary; (3) Multi-stage neural networks behave like a series connection of small individual models. In addition, MSAs at the end of a stage play a key role in prediction. Based on these insights, we propose AlterNet, a model in which Conv blocks at the end of a stage are replaced with MSA blocks. AlterNet outperforms CNNs not only in large data regimes but also in small data regimes. The code is available at https://github.com/xxxnell/how-do-vits-work.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/430bab3890e1e52c4c1f74900b0e408e47a1cb8f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning with Transformers for Image Classification (2022, background): “…have been known to have a tendency to overﬁt training datasets, consequently leading to poor predictive performance in small data regimes, however [23] just showed that this claim is poorly supported, explains the nature of multi-head self-attentions and shows that ViT does not overﬁt even on…”
Wavelet Convolutions for Large Receptive Fields (2024, background): “This discussion complements the analysis that convolutional layers are known to respond to high frequencies in the input [19,43].”
Inception Transformer (2022, background): “This observation also accords with the empirical results in [14], which shows ViT presents the characteristics of low-pass filters.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “However, There have been a few related studies [57] that have demonstrated that transformer blocks have a tendency to deteriorate high-frequency information, such as information about the local texture of objects, to some extent.”
A Survey of Visual Transformers (2021, background): “Slack bias offers visual Transformer a higher upper bound than CNNs based on sufficient training data without sophistic assumptions but performs inferiority and slow convergency in small datasets [208].”
Image De-Raining Transformer (2022, result): “Besides, Fourier transformed features in Section 4.4.1 reveal that self-attention tends to act as a low-passing ﬁlter while CNNs can complement it by preserving some useful high-frequency components, which agrees with recent ﬁndings [108], [109].”
Transformers in Remote Sensing: A Survey (2022, background): “Recent works [2], [3] have shown that ViTs possess content-dependent long-range interaction modeling capabilities and can flexibly adjust their receptive fields to counter nuisances in data and learn effective feature representations.”
Fast Vision Transformers with HiLo Attention (2022, background): “Recent studies have shown that the global attention in MSA helps to capture low frequencies [31].”
Vision Transformers for Single Image Dehazing (2022, background): “We consider that multiplying MHSA is a low-pass filtering, a similar conclusion was presented in a very recent work [29].”
Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, background): “Table 6 shows that the proposed framework greatly increases the sensitivity of sentence embeddings to temporal content whilst better capturing the static content (RadNLI).”
Vision Transformer Adapter for Dense Predictions (2022, background): “Recent studies (Wang et al., 2022a; Wu et al., 2021; Fang et al., 2022; Park & Kim, 2022) show convolutions can help transformers better capture the local spatial information.”
MetaFormer is Actually What You Need for Vision (2021, background): “[41] shows that multi-head self-attentions improve accuracy and generalization by flattening the loss landscapes.”