Visformer: The Vision-friendly Transformer

By Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, Qi Tian. 2021

316 citations recorded by the source corpus.

The past year has witnessed the rapid development of applying the Transformer module to vision problems. While some researchers have demonstrated that Transformer-based models enjoy a favorable ability of fitting data, there are still growing number of evidences showing that these models suffer over-fitting especially when the training data is limited. This paper offers an empirical study by performing step-by-step operations to gradually transit a Transformer-based model to a convolution-based model. The results we obtain during the transition process deliver useful messages for improving visual recognition. Based on these observations, we propose a new architecture named Visformer, which is abbreviated from the ‘Vision-friendly Transformer’. With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy, and the advantage becomes more significant when the model complexity is lower or the training set is smaller. The code is available at https://github.com/danczs/Visformer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, background): “Resonating with our work, [5, 17, 47, 49] present models with convolutional stems, but do not analyze optimizability (our focus).”
RegionViT: Regional-to-Local Attention for Vision Transformers (2021, methodology): “…parallel thread for improving vision transformer is in incorporating CNN-like pyramid structure into ViT (Wu et al., 2021; Wang et al., 2021; Heo et al., 2021; Zhang et al., 2021a; Li et al., 2021; Graham et al., 2021; Liu et al., 2021; Chu et al., 2021a; Yan et al., 2021; Chen et al., 2021b).”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
Aggregating Nested Transformers (2021, background): “Existing reported performance on data efﬁcient benchmarks [10, 22] still far under-perform convnets.”
Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding (2021, background): “Although promising, this increases the supervised training complexity, and existing reported performance on data efficient benchmarks (Hassani et al. 2021; Chen et al. 2021) still significantly underperforms convolutional networks.”
Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion (2021, background): “In addition to these Transformers without convolution operation, several works [52], [53], [54] proposed to blend convolutional layers into Transformers, which further”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, background): “The comparing methods include DeiT (Touvron et al., 2021a), CaiT (Touvron et al., 2021b), LV-ViT (Jiang et al., 2021), CoaT (Xu et al., 2021), Swin (Liu et al., 2021), Twins (Chu et al., 2021), Visformer (Chen et al., 2021), ConViT (d’Ascoli et al., 2021), TNT (Han et al., 2021a), Efﬁcient-Net (Tan ”
MPViT: Multi-Path Vision Transformer for Dense Prediction (2021, background): “7 Base (B) [1, 3, 8, 3] [128, 224, 368, 480] 74.”
A Survey on Vision Transformer (2020, background): “based models can be more difficult to enjoy a favorable ability of fitting data [55], [38], [245], in other words, they are sensitive to the choice of optimizer, hyper-parameter, and the schedule of”