Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
By Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, Gao Huang. 2021
264 citations recorded by the source corpus.
Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of"easy"images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of"hard"ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Efficiency & Compression
Citations in the corpus
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (2022, methodology): “In this respect, it shares the same spirit as the methods in (Graves, 2016; Dehghani et al., 2019; Dong et al., 2019; Schwarzschild et al., 2021; Wang et al., 2021; Wei et al., 2022; Wang et al., 2022).”
- Vision Transformer with Deformable Attention (2022, background): “These attention mechanisms include windowed attention [11, 26], global tokens [6, 21, 32], focal attention [43] and dynamic token sizes [37].”
- FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “With the advent of Vision Transformer [11, 39], self-attention techniques have shown great potential in a variety of vision tasks including image classification [41, 43, 30, 46], semantic segmentation [6, 49], object detection [4, 61, 22], and multi-modal tasks [35, 31].”
- A Survey of Visual Transformers (2021, methodology): “(3) For segmentation, the encoder-decoder Transformer models may unify three segmentation sub-tasks into a mask prediction problem via a set of learnable mask embeddings [31], [105], [199].”
- Experts Collaboration Learning for Continual Multi-Modal Reasoning (2023, background): “Some attempts [55], [56], [57], [58], [59], [60] have been made to achieve dynamic and flexible network structures, and a considerable part of existing works focus on utilizing dynamic network structures to save computation resources and enhance model efficiency.”