Do Vision Transformers See Like Convolutional Neural Networks?

By Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy. 2021

1,440 citations recorded by the source corpus.

Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets (2022, background): “This result corroborates recent results in neural architecture literature, which find that supervised and self-supervised representations are similar [Grigg et al. 2021], whereas ViTs and CNNs learn different representations [Raghu et al. 2021].”
DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning (2022, background): “This observation also fits the intuition that different layers in deep learning models capture different types of knowledge [49,66], and thus naturally fit different prompts.”
ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, background): “The previous studies [44, 45] find that the attention maps across adjacent layers of vision transformer exhibit very high similarity.”
ViM: Out-Of-Distribution with Virtual-logit Matching (2022, background): “We hypothesize that it is related to the observation in [28] that higher layers of ViT maintain spatial location information more faithfully than ResNets.”
MetaFormer is Actually What You Need for Vision (2021, background): “[42] compare the feature difference between ViT and CNNs, finding that self-attention enables early aggregation of global information while residual connections strongly propagate features from lower to higher layers.”
StyTr2: Image Style Transfer with Transformers (2021, background): “Second, the transformer architecture models relationships in input shapes [41], and different layers extract similar structural information [46] (see Fig.”
Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis (2021, background): “In comparison to CNN-based counterparts, transformerbased models learn stronger features representations during pre-training, and as a result perform favorably on fine-tuning downstream tasks [38].”
Do Self-Supervised and Supervised Methods Learn Similar Visual Representations? (2021, methodology): “Prior work has demonstrated the utility of Centered Kernel Alignment (CKA) as a similarity index which elegantly addresses these challenges [16], enabling the analysis of a variety of neural architectures [16, 17, 18].”
Auto-scaling Vision Transformers without Training (2022, background): “Second, the learning behaviors of ViT, including (loss of) feature diversity (Zhou et al., 2021), receptive ﬁelds (Raghu et al., 2021) and augmentations (Touvron et al., 2020; Jiang et al., 2021), differ vastly from CNNs. Beneﬁting from self-attention, ViT can capture global information even with…”
How Do Vision Transformers Work? (2022, background): “As Raghu et al. (2021) pointed out, ViTs have a uniform representations across all layers.”
Multimodal Learning With Transformers: A Survey (2022, background): “, its internal representation robustness [99], the continuous behaviour of its latent representation propagation [100], [101].”
Vision-Language Pre-Training with Triple Contrastive Learning (2022, background): “We suspect that the difference between features learned by CNNs and vision transformers lead to this observation [37].”
CellViT: Vision Transformers for Precise Cell Segmentation and Classification (2023, background): “Vision Transformers have shown promising results not only in image classification [50, 51, 52], but also in other vision tasks such as object detection [53] and semantic segmentation [20, 5].”
TransMorph: Transformer for unsupervised medical image registration (2021, neutral): “Hence, Transformer may fall short of delivering high-resolution feature maps and aggregating local information at lower layers (Raghu et al., 2021).”
Inception Transformer (2022, background): “Moreover, we find that lower layers often need more local information, while higher layers desire more global information, which also accords with the observations in [27].”
Deep ViT Features as Dense Visual Descriptors (2021, background): “Concurrently, [42, 11, 37] study theoretical aspects of the underlying machinery, aiming to analyze how ViTs process visual data compared to CNN models.”
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, background): “Motivated by recent studies on the importance of shortcuts in ViTs [24], we add a channel-wise projector as a diversity-enhanced shortcut (DES).”
Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, neutral): “Recently, Mamba [4] has demonstrated its efficiency and effectiveness in long-range dependency modeling.”
MAT: Mask-Aware Transformer for Large Hole Image Inpainting (2022, background): “First, the incorporation of local inductive priors in early visual processing remains vital for better representation [43] and optimizability [60].”