Fast Vision Transformers with HiLo Attention

By Zizheng Pan, Jianfei Cai, Bohan Zhuang. Neural Information Processing Systems, 2022

296 citations recorded by the source corpus.

Vision Transformers (ViTs) have triggered the most recent and significant breakthroughs in computer vision. Their efficient designs are mostly guided by the indirect metric of computational complexity, i.e., FLOPs, which however has a clear gap with the direct metric such as throughput. Thus, we propose to use the direct speed evaluation on the target platform as the design principle for efficient ViTs. Particularly, we introduce LITv2, a simple and effective ViT which performs favourably against the existing state-of-the-art methods across a spectrum of different model sizes with faster speed. At the core of LITv2 is a novel self-attention mechanism, which we dub HiLo. HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on global structures, whereas a multi-head self-attention layer neglects the characteristic of different frequencies. Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the average-pooled low-frequency keys and values from each window and each query position in the input feature map. Benefiting from the efficient design for both groups, we show that HiLo is superior to the existing attention mechanisms by comprehensively benchmarking FLOPs, speed and memory consumption on GPUs and CPUs. For example, HiLo is 1.4x faster than spatial reduction attention and 1.6x faster than local window attention on CPUs. Powered by HiLo, LITv2 serves as a strong backbone for mainstream vision tasks including image classification, dense detection and segmentation. Code is available at https://github.com/ziplab/LITv2.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/bf6ce546c589fa8054b3972b266532664914bd21

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization (2023, methodology): “From Table 7, our model is highly competitive to RVT and ConvNeXt, in fact FastViT-M36 has better clean accuracy, better robustness to corruptions and similar out-of-distribution robustness as ConvNeXt-S which has 6.1M more parameters and has 10% more FLOPs than our model.”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, background): “Swin [52] and following works [20, 51] propose window-based attention such that the receptive field is constrained to a pre-defined window size, which also inspires subsequent work to refine attention patterns [11, 78, 81, 61].”
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention (2023, methodology): “Although there are some methods proposed to address this issue by simplifying the computation of standard softmax self-attention, e.g. , sparse attention [34, 57, 61, 75] and low-rank approximation [11,51,74], they often come at the cost of accuracy degradation and limited acceleration.”