I-BERT: Integer-only BERT Quantization

By Sehoon Kim, A. Gholami, Zi-Mei Yao, Michael W. Mahoney, Kurt Keutzer. 2021

459 citations recorded by the source corpus.

Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has been open-sourced.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “Further work includes generalizing the approach other tasks and experimenting with complementary compression methods such as pruning and quantization (Kim et al., 2021; Sanh et al., 2020), as well as increasing linguistic coverage (Joshi et al., 2020).”
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022, background): “It proves to be effective for various convolutional neural works (CNNs) (Han et al., 2016; Jacob et al., 2018; Nagel et al., 2019; Wang et al., 2019; Lin et al., 2020) and transformers (Shen et al., 2020; Kim et al., 2021; Liu et al., 2021; Wang et al., 2020; Bondarenko et al., 2021).”
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from different aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “Kim et al. (2021) proposed a fully integer-only arithmetic inference scheme based on second-order polynomial approximations for GELU, Softmax, and LayerNorm non-linearities.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Recently, [259] leverages mix-precision quantization to achieve 23% speedup for ResNet50, as compared to INT8 inference without accuracy degradation, and [130] extends INT8-only inference to BERT model to enable up to 4.0 faster inference than FP32.”
Learned Token Pruning for Transformers (2021, unknown): “However, there are several challenges to consider.”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “Existing quantization methods can be roughly divided into two categories according to the quantization bitwidth, namely, fixed-point quantization [4,7,9,19,22,25,28,32,46,72,75,77] and binary quantization [1, 26, 41, 45, 47, 59, 60].”
SqueezeLLM: Dense-and-Sparse Quantization (2023, methodology): “This can be broadly categorized into efficient architecture design [32, 38, 42, 60, 67, 73], pruning [18, 20, 40, 41, 48, 52, 65, 79], neural architecture search [7, 58, 59, 66, 74, 78], and quantization [35, 55, 81, 82].”
Rethinking Vision Transformers for MobileNet Size and Speed (2022, methodology): “As applicable to CNNs, architecture search, pruning, and quantization techniques are also thoroughly investigated [36, 37, 54, 38, 9, 47, 50].”
Big Little Transformer Decoder (2023, methodology): “Well-known approaches include efficient architecture design [25, 31, 34, 57, 64, 72], quantization [8, 29, 51, 71, 76, 78, 79], pruning [11, 13, 32, 33, 39, 46, 61], and neural architecture search [5, 53, 54, 63, 73, 77].”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “…including pruning (Wang et al., 2020; Gordon et al., 2020), low-rank factorization (Ma et al., 2019; Noach and Goldberg, 2020; Lan et al., 2020), quantization (Zafrir et al., 2019; Shen et al., 2020; Kim et al., 2021), knowledge distillation (Jiao et al., 2020; Sanh et al., 2019; Sun et al., 2020)”
Extreme Compression for Pre-trained Transformers Made Simple and Efficient (2022, methodology): “Quantization becomes practically appealing as it not only reduces memory bandwidth consumption but also brings the additional benefit of further accelerating inference on supporting hardware [44, 24, 12].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, methodology): “…eliminate ﬂoating-point operations from the quantized model, integer-only quantization techniques emulate the full-precision multiplication by 32-bit integer multiplication followed by bit shifting (Jacob et al., 2018; Zhu et al., 2020; Wu et al., 2020; Yao et al., 2021; Kim et al., 2021).”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “[25] approximates nonlinear operations to implement integer-only quantization.”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “…language models, involving techniques such as distillation (Hin-ton et al., 2015), quantization (Hubara et al., 2018; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020; Kur-tic et al., 2022; Voita et al., 2019),…”
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024, background): “…significant work (Liu et al., 2023b) focused on accelerating LLM inference, such as low-bit quantization (Hubara et al., 2018; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020), and knowledge distillation (Hinton et…”