Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations

By Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Pengtao Xie. arXiv.org, 2022

457 citations recorded by the source corpus.

Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Complete leverage of these image tokens brings redundant computations since not all the tokens are attentive in MHSA. Examples include that tokens containing semantically meaningless or distractive image backgrounds do not positively contribute to the ViT predictions. In this work, we propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training. For each forward inference, we identify the attentive image tokens between MHSA and FFN (i.e., feed-forward network) modules, which is guided by the corresponding class token attention. Then, we reorganize image tokens by preserving attentive image tokens and fusing inattentive ones to expedite subsequent MHSA and FFN computations. To this end, our method EViT improves ViTs from two perspectives. First, under the same amount of input image tokens, our method reduces MHSA and FFN computation for efficient inference. For instance, the inference speed of DeiT-S is increased by 50% while its recognition accuracy is decreased by only 0.3% for ImageNet classification. Second, by maintaining the same computational cost, our method empowers ViTs to take more image tokens as input for recognition accuracy improvement, where the image tokens are from higher resolution images. An example is that we improve the recognition accuracy of DeiT-S by 1% for ImageNet classification at the same computational cost of a vanilla DeiT-S. Meanwhile, our method does not introduce more parameters to ViTs. Experiments on the standard benchmarks show the effectiveness of our method. The code is available at https://github.com/youweiliang/evit

Subfields: Computer Vision and Pattern Recognition, Neurology

View Paper: https://www.semanticscholar.org/paper/625270a54be430ad6262f848babb0d106af5b183

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Neurology

Citations in the corpus

Token Merging: Your ViT But Faster (2022, background): “A promising subfield of ViTs have recently emerged where, due to the input-agnostic nature of transformers, tokens can be pruned at runtime to enable a faster model (Rao et al., 2021; Yin et al., 2022; Meng et al., 2022; Liang et al., 2022; Kong et al., 2022).”
AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition (2022, background): “The strong capability of modeling long-range relation has facilitated Transformer in various vision tasks, including image classification [31, 63, 60], object detection [11, 98, 22], semantic/instance segmentation [86], video understanding [8, 2, 33, 57], point cloud modeling [95, 41], 3D Object Rec”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, unknown): “In this section, we first review the basic implementation of large mutilmodal models (LMMs), with a particular focus on the visual encoder component ( i.e ., Vision Transformer).”
Scaling Open-Vocabulary Object Detection (2023, background): “Various forms of token dropping or pooling have been proposed to improve efficiency [19, 26, 35, 20, 2].”
Dilated Neighborhood Attention Transformer (2022, background): “Their restriction of self attention can potentially improve convergence by avoiding self attention’s possible redundant interactions, such as those with repetitive, background, or distracting tokens [23, 33].”