AdaViT: Adaptive Vision Transformers for Efficient Image Recognition

By Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, Ser-Nam Lim. Computer Vision and Pattern Recognition, 2021

333 citations recorded by the source corpus.

Built on top of self-attention mechanisms, vision transformers have demonstrated remarkable performance on a variety of tasks recently. While achieving excellent performance, they still require relatively intensive computational cost that scales up drastically as the numbers of patches, self-attention heads and transformer blocks increase. In this paper, we argue that due to the large variations among images, their need for modeling long-range dependencies between patches differ. To this end, we introduce AdaViT, an adaptive computation framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use throughout the backbone on a per-input basis, aiming to improve inference efficiency of vision transformers with a minimal drop of accuracy for image recognition. Optimized jointly with a transformer backbone in an end-to-end manner, a light-weight decision network is attached to the backbone to produce decisions on-the-fly. Extensive experiments on ImageNet demonstrate that our method obtains more than 2 × improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy, achieving good efficiency/accuracy trade-offs conditioned on different computational budgets. We further conduct quantitative and qualitative analysis on learned usage polices and provide more insights on the redundancy in vision transformers. Code is available at ht tps: / / gi thub. com/MengLcool/AdaVi T.

View Paper: https://www.semanticscholar.org/paper/38212997a6e8c55141574c329bb58d2eadcb0db5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jianwei Yang
2. Jianfeng Gao
3. Zuxuan Wu
4. Lu Yuan
5. Zhe Gan
6. Chunyuan Li
7. Christoph Feichtenhofer
8. Xiyang Dai

Citations in the corpus

• Focal Modulation Networks (2022, background): “To address this, many works have proposed SA variants through token coarsening [69], window attention [46, 65, 87], dynamic token selection [51, 81, 50], or the hybrid [79, 14].”
• EfficientFormer: Vision Transformers at MobileNet Speed (2022, background): “Following this success, several efforts explore the potential of ViT by improving training strategies [3, 4, 5], introducing architecture changes [6, 7], redesigning attention mechanisms [8, 9], and elevating the performance of various vision tasks such as classification [10, 11, 12], segmentation…”
• Token Merging: Your ViT But Faster (2022, background): “A promising subfield of ViTs have recently emerged where, due to the input-agnostic nature of transformers, tokens can be pruned at runtime to enable a faster model (Rao et al., 2021; Yin et al., 2022; Meng et al., 2022; Liang et al., 2022; Kong et al., 2022).”
• BEVT: BERT Pretraining of Video Transformers (2021, background): “Motivated by the impressive performance of transformers in a wide range of NLP tasks, there is a growing interest in using transformers for computer vision tasks [19,20,28,36,38,43,49,56,73].”