Early Convolutions Help Transformers See Better

By Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Doll'ar, Ross B. Girshick. 2021

950 citations recorded by the source corpus.

Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p*p convolution (p=16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3*3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ~1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, methodology): “We use the regularizations listed in Table 11 and disable others, following [64].”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “In the meantime, as suggested by [13], [86], early convolutions help the stability on the training of ViT, so we further use a “Conv Steam” with 4 convolutional layers as our patch embedding module to build our VOLO model.”
Scalable Diffusion Models with Transformers (2022, background): “DiT retains many of the best practices of ViTs.”
A ConvNet for the 2020s (2022, background): “The original ViT [20] first studied a hybrid configuration, and a large body of follow-up works focused on reintroducing convolutional priors to ViT, either in an explicit [15, 16, 21, 82, 86, 88] or implicit [45] fashion.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [14,21,34,42,63,68,71,75,77,78,82].”
Improving Vision Transformers for Incremental Learning (2021, methodology): “Fol- lowing LwF, Aljundi et al. propose gating autoencoders to automatically select the task classifier for each testing sample [4].”
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (2022, methodology): “Feature Extraction: At the beginning, we extract the shallow features F H SE ( · ) from the source images I 1 and I 2 , which can be expressed as: The convolutional layers are good at early visual processing, resulting in more stable optimization and better results [74].”
Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation (2021, methodology): “We do not use attention operations in the stem since early convolutions are more effective to extract low-level features than self-attentions [14,32].”
A Survey of Visual Transformers (2021, background): “• The convolutional patchify stem (ViTc [59]) and early convolutional stage (CoAtNet [40]) can significantly boost the accuracy of the Transformers, especially for large models.”
M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection (2021, background): “The intuition of using convolutions here is to ensure faster convergence and more stable training [66].”
Activating More Pixels in Image Super-Resolution Transformer (2022, background): “Besides, many works illustrate that convolution can help Transformer get better visual representation or achieve easier optimization [26,59,61,63,71].”
MaxViT: Multi-Axis Vision Transformer (2022, methodology): “A broad range of works fall into this category, either explicitly hybridized [16,20,21,89,90,92] or in an implicit fashion [12,54].”
DeiT III: Revenge of the ViT (2022, neutral): “One difficult question to address is whether the improved performance is due to a specific architectural design, or because it facilitates the optimization as suggested it is the case for convolutions with ViTs [60].”