LocalViT: Analyzing Locality in Vision Transformers

By Yawei Li, Kai Zhang, Jie Cao, Radu Timofte, Luc Van Gool. 2021

448 citations recorded by the source corpus.

The aim of this paper is to study the influence of locality mechanisms in vision transformers. Transformers originated from machine translation and are particularly good at modelling long-range dependencies within a long sequence. Although the global interaction between the token embeddings could be well modelled by the self-attention mechanism of transformers, what is lacking is a locality mechanism for infor-mation exchange within a local region. In this paper, locality mechanism is systematically investigated by carefully designed controlled experiments. We add locality to vision transformers into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks. The importance of locality mechanisms is validated in two ways: 1) A wide range of design choices (activation function, layer placement, expansion ratio) are available for incorporating locality mechanisms and proper choices can lead to a performance gain over the baseline, and 2) The same locality mechanism is successfully applied to vision transformers with different architecture designs, which shows the generalization of the locality concept. For ImageNet2012 classification, the locality-enhanced transformers outperform the baselines Swin-T [1], DeiT-T [2] and PVT-T [3] by 1.0%, 2.6 % and 3.1 % with a negligible increase in the number of parameters and computational effort. Code is available at https://github.com/ofsoundof/LocalViT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Model Architectures

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “LocalViT [23] and CeiT [50] introduce connectivity of local regions into ViTs by adding a depth-wise convolution in feed-forward networks (FFN).”
Transformers in Vision: A Survey (2021, background): “LocalViT [123] introduces depthwise convolutions to enhance local features modeling capability of ViTs.”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [8, 14, 23, 31, 50, 55, 58, 62, 64, 65, 67].”
Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “The SA mechanism in transformers can effectively model the dependency across data, and it has achieved impressive results on several high-level vision tasks, such as image classification [13, 26, 34, 35, 44, 52], image detection [8, 32, 35, 52], and segmentation [7, 35, 57, 63].”
Activating More Pixels in Image Super-Resolution Transformer (2022, methodology): “A series of Transformer-based methods [13,27,56,51,36,18,12,6,57,60,25,42] have been developed for high-level vision tasks, including image classification [36,13,27,44,49], object detection [34,48,36,4,6], segmentation [55,51,16,2], etc.”
MISSFormer: An Effective Transformer for 2D Medical Image Segmentation (2022, methodology): “Second, the transformer lacks the ability to extract the local context [30], [31], [32], although Uformer, SegFormer and PVTv2 tried to overcome the limitation by embedding a convolutional layer in feed-forward network directly, we argue that this approach limits the discrimination of features,…”
Filter-enhanced MLP is All You Need for Sequential Recommendation (2022, background): “Recently, several studies question the necessity of CNN [54] and self-attention network [28, 29], and propose to use MLP to replace the above architectures.”
A Survey on Vision Transformer (2020, methodology): “CvT [241], CeiT [259], LocalViT [132] and CMT [77] analyzed the potential drawbacks when directly borrowing Transformer architectures from NLP and combined the convolutions with transformers together.”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
Efficient Training of Visual Transformers with Small Datasets (2021, methodology): “To alleviate this problem, a second generation of VTs has very recently been independently proposed by different groups [61, 35, 58, 60, 59, 33, 28].”
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration (2023, methodology): “Inspired by the success in natural language processing, transformers have been applied in both visual perception tasks [5, 15, 48, 73, 79, 98] and image restoration tasks [6].”
Vision Transformer for Small-Size Datasets (2021, background): “For example, some adopted a hierarchical structure like CNNs to leverage various receptive ﬁelds [12, 24, 39], and the others tried to modify the self-attention mechanism itself [22,24,34,39,40].”
Aggregating Nested Transformers (2021, background): “…mechanism [41, 51] attract wide interests in recent computer vision community, with explorations of more suitable architectural design that can learn visual representation effectively, such as injecting convnet elements [33, 46, 55], building local or hierarchical structures [54, 62].”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, background): “3) introduce convolution to shuffle transformer block [25, 40].”
Vision Transformers for Single Image Dehazing (2022, methodology): “Uformer [60] uses Swin Transformer blocks to build a U-Net-like network and inserted depth-wise convolution (DWConv) [61] in the feed-forward network (FFN) like LocalViT [62].”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “Some following works such as T2T-ViT [55], TNT [17], CaiT [39], DeepViT [62], CPVT [6], LocalViT [28] and CrossViT [5] focus on improving the architecture design of ViT.”
Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, methodology): “We also compare our method with several highly competitive baselines, i.e., TNT [16], LocalViT [27], CrossViT [5], PVT [41], ViT [11] and ResNet [18].”