Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer

By Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, Xing Sun. AAAI Conference on Artificial Intelligence, 2021

301 citations recorded by the source corpus.

Vision transformers (ViTs) have recently received explosive popularity, but the huge computational cost is still a severe issue. Since the computation complexity of ViT is quadratic with respect to the input sequence length, a mainstream paradigm for computation reduction is to reduce the number of tokens. Existing designs include structured spatial compression that uses a progressive shrinking pyramid to reduce the computations of large feature maps, and unstructured token pruning that dynamically drops redundant tokens. However, the limitation of existing token pruning lies in two folds: 1) the incomplete spatial structure caused by pruning is not compatible with structured spatial compression that is commonly used in modern deep-narrow transformers; 2) it usually requires a time-consuming pre-training procedure. To tackle the limitations and expand the applicable scenario of token pruning, we present Evo-ViT, a self-motivated slow-fast token evolution approach for vision transformers. Specifically, we conduct unstructured instance-wise token selection by taking advantage of the simple and effective global class attention that is native to vision transformers. Then, we propose to update the selected informative tokens and uninformative tokens with different computation paths, namely, slow-fast updating. Since slow-fast updating mechanism maintains the spatial structure and information flow, Evo-ViT can accelerate vanilla transformers of both flat and deep-narrow structures from the very beginning of the training process. Experimental results demonstrate that our method significantly reduces the computational cost of vision transformers while maintaining comparable performance on image classification. For example, our method accelerates DeiT-S by over 60% throughput while only sacrificing 0.4% top-1 accuracy on ImageNet-1K, outperforming current token pruning methods on both accuracy and efficiency.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/d045133e6e022684329ff944d67f91888be1bc3b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Fan Tang
2. Rong Jin
3. Changsheng Xu
4. Chongyang Ma
5. Weihua Chen
6. Fan Wang
7. Lei Wang
8. Xingjia Pan

In Electrical and Electronic Engineering

1. Rong Jin
2. Changsheng Xu

Citations in the corpus

• StyTr2: Image Style Transfer with Transformers (2021, methodology): “Inspired by the breakthrough of transformer in NLP, many researchers have developed vision transformers for various image/video related tasks [60], including object detection [6, 12, 65], semantic segmentation [53, 64], image classification [10, 16, 37, 55, 61], image processing and generation [8, 1”
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al. 2021a; Wang et al. 2021b; Han et al. 2021a; Chen et al. 2021; Ranftl, Bochkovskiy, and Koltun 2021; Liu et al. 2021; Chen, Fan, and Panda 2021; Zhang et al. 202”