ConViT: improving vision transformers with soft convolutional inductive biases

By Stéphane d'Ascoli, Hugo Touvron, Matthew L. Leavitt, Ari S. Morcos, Giulio Biroli, Levent Sagun. 2021

1,066 citations recorded by the source corpus.

Convolutional architectures have proven to be extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision transformers rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ‘soft’ convolutional inductive bias. We initialize the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT (Touvron et al 2020 arXiv:2012.12877) on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analyzing how it has escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, background): “This perspective resonates with the findings of [9], who observe that early transformer blocks prefer to learn more local attention patterns than later blocks.”
Towards Robust Vision Transformer (2021, methodology): “ConViT [6], Swin [25] and LeViT [12] both use more self-attention heads and smaller dimensions of keys and queries to achieve better performance at a controllable FLOPs.”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “(9) suggests a very elegant idea of initializing Vision Transformer, such that it behaves similarly to convolutional neural network in the beginning of training.”
S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, background): “The original backbone of DyTox is a shallow ConViT [15] ( 5 self-attention blocks and a task-attention block) without pre-training.”
A ConvNet for the 2020s (2022, background): “The original ViT [20] first studied a hybrid configuration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an explicit [15, 16, 21, 82, 86, 88] or implicit [45] fashion.”
XCiT: Cross-Covariance Image Transformers (2021, background): “…exploiting pixel-level information and integrating with an inter-patch transformer to attain higher representation power. d’Ascoli et al. [18] consider the initialization of self-attention blocks as a convolutional operator, and demonstrate that such initialization improves the…”
Recipe for a General, Powerful, Scalable Graph Transformer (2022, methodology): “Considering the great successes of Transformers in natural language processing (NLP) [52, 30] and recently also in computer vision [16, 23, 22], it is natural to study their applicability in the graph domain as well.”
Deep Class-Incremental Learning: A Survey (2023, methodology): “For ViT-based methods like DyTox and L2P, we follow the original implementation and use ConViT [201] for DyTox and pre-trained ViT-B/16 [24] for L2P.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, background): “Other recent related work has looked at designing hybrid ViT-CNN models [49, 11], drawing on structural differences between the models.”
Language Models are Few-shot Learners for Prognostic Prediction (2023, background): “Although transformers have advantages in modeling high-dimensional tabular data thanks to the capacity of long-distance dependency modeling, their efﬁcacy can still be hampered when labeled data is scarce given the nature of data-hungry and low inductive bias (d’Ascoli et al., 2021).”
MetaFormer is Actually What You Need for Vision (2021, background): “A lot of works have been focusing on improving the token mixing approach of transformers by shifted windows [35], relative position encoding [59], refining attention map [66], or incorporating convolution [12, 20, 58], etc.”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, background): “The transformer has achieved promising success in image classification (Dosovitskiy et al. 2021; Touvron et al. 2021; D’Ascoli et al. 2021), object detection (Carion et al.”
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications (2022, background): “3 265 μs 2: Conv=[3, 3, 9, 2], SDTA=[0, 0, 0, 1] 78.”
Patches Are All You Need? (2022, background): “, 2019) and can be initialized or regularized to be like it (d’Ascoli et al., 2021); other works simply add convolution operations to transformers (Dai et al.”
GMFlow: Learning Optical Flow via Global Matching (2021, background): “The top-performing sparse methods (e.g., SuperGlue [34] and LoFTR [38]) adopt Transformers [41] to reason about the mutual relationship between feature descriptors, and the correspondences are extracted with an explicit matching layer (e.g., softmax layer [43]).”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, neutral): “The comparing methods include DeiT (Touvron et al., 2021a), CaiT (Touvron et al., 2021b), LV-ViT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021), Twins (Chu et al., 2021), Visformer (Chen et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021a), Efﬁcient-Net (Tan ”
HRFormer: High-Resolution Transformer for Dense Prediction (2021, methodology): “They have shown that increasing the spatial resolution of the representations output by the Vision Transformer is important for semantic segmen- tation.”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Recently, some works try to introduce the intrinsic IB of CNNs into vision transformers explicitly [26,58,25,43,18,86, 80,91,9,49].”
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions (2022, background): “Many enhanced vision Transformers also use lightweight convolutions in the basic building block to efficiently capture neighboring patterns [15, 56, 17] or relax the quadratic complexity of self-attention [9, 54, 18].”