DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification

By Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, Cho-Jui Hsieh. 2021

1,174 citations recorded by the source corpus.

Attention is sparse in vision transformers. We observe the final prediction in vision transformers is only based on a subset of most informative tokens, which is sufficient for accurate image recognition. Based on this observation, we propose a dynamic token sparsification framework to prune redundant tokens progressively and dynamically based on the input. Specifically, we devise a lightweight prediction module to estimate the importance score of each token given the current features. The module is added to different layers to prune redundant tokens hierarchically. To optimize the prediction module in an end-to-end manner, we propose an attention masking strategy to differentiably prune a token by blocking its interactions with other tokens. Benefiting from the nature of self-attention, the unstructured sparse tokens are still hardware friendly, which makes our framework easy to achieve actual speed-up. By hierarchically pruning 66% of the input tokens, our method greatly reduces 31%~37% FLOPs and improves the throughput by over 40% while the drop of accuracy is within 0.5% for various vision transformers. Equipped with the dynamic token sparsification framework, DynamicViT models can achieve very competitive complexity/accuracy trade-offs compared to state-of-the-art CNNs and vision transformers on ImageNet. Code is available at https://github.com/raoyongming/DynamicViT

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

EfficientFormer: Vision Transformers at MobileNet Speed (2022, methodology): “To accelerate ViT, many approaches have been introduced with different methodologies, such as proposing new architectures or modules [53, 54, 55, 56, 57, 58], re-thinking self-attention and sparse-attention mechanisms [59, 60, 61, 62, 63, 64, 65], and utilizing search algorithms that are widely expl”
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding (2023, methodology): “There have also been recent methods [6, 39, 49, 63, 68] to explore the role of dynamic token within the transformer framework.”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al. 2021a; Wang et al. 2021b; Han et al. 2021a; Chen et al. 2021; Ranftl, Bochkovskiy, and Koltun 2021; Liu et al. 2021; Chen, Fan, and Panda 2021; Zhang et al. 202”
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, methodology): “One related topic is the adaptive inference [28,37,48] in vision transformers, which is proposed to accelerate the computation of ViT.”
Spikformer: When Spiking Neural Network Meets Transformer (2022, methodology): “There are some methods aiming to reduce the computational complexity of self-attention or improve its ability of modeling visual dependencies (Song, 2021; Yang et al., 2021; Rao et al., 2021; Choromanski et al., 2020).”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “…training from scratch, a significant accuracy reduction is observed in DynamicViT, which suggests that DynamicViT requires a teacher ViT (i.e., the parameter initialization, the distillation loss, and the prediction (KL) loss from a pretrained ViT (Rao et al., 2021)) to obtain a good performance.”
Global Filter Networks for Image Classification (2021, background): “The transformer architecture, originally designed for the natural language processing (NLP) tasks [42], has shown promising performance on various vision problems recently [10, 40, 27, 49, 4, 47, 35, 5].”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, background): “producing policies on what patch size [47] and which patches [30, 33] to use conditioned on input image.”
Transformers in Medical Imaging: A Survey (2022, background): “Recently, numerous efforts have been made to reduce complexity, including sparse attention [85], linearization attention [86], low-rank attention [87], memory compression based approaches [88], and improved MHSA [89].”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, methodology): “For pruning methods (Rao et al. 2021; Tang et al. 2021), by pruning η% tokens, at least η% FLOPS in the FFN and MSA modules can be reduced.”
Focal Modulation Networks (2022, background): “To address this, many works have proposed SA variants through token coarsening [69], window attention [46, 65, 87], dynamic token selection [51, 81, 50], or the hybrid [79, 14].”
MaxViT: Multi-Axis Vision Transformer (2022, background): “More recent works have been focused on improving model and data efficiency, including sparse attention [1,19,55,76,85,88], improved locality [24,90], pyramidal designs [22,77, 86], improved training strategies [3,71,72,94], etc.”
Scaling Open-Vocabulary Object Detection (2023, background): “Various forms of token dropping or pooling have been proposed to improve efficiency [19, 26, 35, 20, 2].”
Token Merging: Your ViT But Faster (2022, methodology): “4, we compare ToMe to token pruning works that use DeiT-S training3: A-ViT (Yin et al., 2022), DynamicViT (Rao et al., 2021), and SP-ViT (Kong et al., 2022) with throughput measured on a V100.”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “Table 4: Comparisons between DVT and three state-of-the-art methods for improving the computational efﬁciency of vision Transformers, i.e., Data distillation [38], DynamicViT [32] and Patch slimming [36].”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, background): “Some other works [40, 11, 51] also make meaningful efforts on vision transformer efficiency.”