Post-Training Quantization for Vision Transformer

By Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, Wen Gao. 2021

505 citations recorded by the source corpus.

Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29\% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Yunhe Wang
3. Luc Van Gool
4. Lu Yuan
5. Kai Han
6. Jianyuan Guo
7. Jiashi Feng
8. Ming-Hsuan Yang

In Efficiency & Compression

1. Yunhe Wang
2. Ji Lin
3. Song Han
4. Dan Alistarh
5. Kai Han
6. Dacheng Tao
7. Chunjing Xu
8. Zi-Mei Yao

Citations in the corpus

• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
• Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “Popular quantization methods include post-training quantization [45, 33, 29], quantization-aware training (QAT) [23, 4, 31, 22], and their variations [44, 24, 12].”
• The YOLO Framework: A Comprehensive Review of Evolution, Applications, and Benchmarks in Object Detection (2024, unknown): “YOLOv9’s PGI ensured smoother backpropagation across multiple prediction branches, significantly improving convergence and overall detection accuracy.”
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, unknown): “One promising avenue for continual learning is key-value methods, stemming from computer vision [26, 34, 42] For example, recent works have demonstrated continual prompt-learning for NLP [44, 45] for applications like text retrieval [47].”
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers (2022, methodology): “It is notable that our work is orthogonal to existing literature on model compression techniques such as quantization [44,36,26,44] and pruning [85,72,73,39].”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “Quantization: quantization Liu et al. (2021); Gholami et al. (2022); Dettmers et al. (2022, 2024) is a widely accepted technique in the field of model compression, which can significantly save the storage and computational costs of deep learning models.”
• Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “Similar to CNNs, architecture search [12, 24, 9, 92, 50, 17, 80], pruning [88], and quantization [54] are also explored for ViTs.”
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “[14] Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao.”
• A Survey on Vision Transformer (2020, background): “[150] explored a post-training quantization scheme to reduce the memory storage and computational costs of vision transformers.”