A-ViT: Adaptive Tokens for Efficient Vision Transformer

By Hongxu Yin, Arash Vahdat, José Manuel Álvarez, Arun Mallya, Jan Kautz, Pavlo Molchanov. 2021

538 citations recorded by the source corpus.

We introduce A - ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A - ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds. We refor-mulate Adaptive Computation Time (ACT [17]) for this task, extending halting to discard redundant spatial tokens. The appealing architectural properties of vision transformers enables our adaptive token reduction mechanism to speed up inference without modifying the network architecture or inference hardware. We demonstrate that A - ViT requires no extra parameters or sub-network for halting, as we base the learning of adaptive halting on the original network parameters. We further introduce distributional prior regularization that stabilizes training compared to prior ACT approaches. On the image classification task (ImageNet1K), we show that our proposed A - ViT yields high efficacy in filtering informative spatial features and cutting down on the overall compute. The proposed method improves the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3% accuracy drop, outperforming prior art by a large margin.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Efficiency & Compression

Citations in the corpus

Scalable Adaptive Computation for Iterative Generation (2022, background): “, 2017) or sparse computation (Yin et al., 2021), facing the challenges nondifferentiability and dynamic or masked computation graphs.”
Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework (2022, background): “One related topic is the adaptive inference [28,37,48] in vision transformers, which is proposed to accelerate the computation of ViT.”
Token Merging: Your ViT But Faster (2022, methodology): “4, we compare ToMe to token pruning works that use DeiT-S training3: A-ViT (Yin et al., 2022), DynamicViT (Rao et al., 2021), and SP-ViT (Kong et al., 2022) with throughput measured on a V100.”
Focal Modulation Networks (2022, background): “To address this, many works have proposed SA variants through token coarsening [69], window attention [46, 65, 87], dynamic token selection [51, 81, 50], or the hybrid [79, 14].”
LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, unknown): “Uni-Modal token reduction methods [4, 13, 17, 25, 31, 47] primarily focus on accelerating ViT computation speed by progressively reducing token numbers across trans-former blocks, thereby lowering the internal ViT computational cost.”
Scaling Open-Vocabulary Object Detection (2023, background): “Various forms of token dropping or pooling have been proposed to improve efficiency [19, 26, 35, 20, 2].”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “For the second concern, an intuitive approach is to drop the similar tokens [42, 43, 33, 44] or, like PNP, apply dropping rate schedulers.”