BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

By Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fisher Yu, Shi Gu. 2021

675 citations recorded by the source corpus.

We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a good balance between cross-layer dependency and generalization error. To further employ the power of quantization, the mixed precision technique is incorporated in our framework by approximating the inter-layer and intra-layer sensitivity. Extensive experiments on various handcrafted and searched neural architectures are conducted for both image classification and object detection tasks. And for the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 comparable with QAT and enjoy 240 times faster production of quantized models. Codes are available at https://github.com/yhhhli/BRECQ.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022, methodology): “Further, we estimate the runtime of roundingbased methods [20, 17], which are significantly more accurate, but typically perform 10× more SGD steps.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “2) Post-Training Quantization: An alternative to the expensive QAT method is Post-Training Quantization (PTQ) which performs the quantization and the adjustments of the weights, without any ﬁne-tuning [11, 24, 40, 60, 61, 68, 69, 88, 107, 140, 146, 170, 178, 273].”
Sharpness-aware Quantization for Deep Neural Networks (2021, methodology): “We compare with enormous fixedpoint quantization methods, including DoReFa-Net [75], PACT [9], LQ-Nets [72], DSQ [21], FAQ [51], QIL [28], Auxi [76], PROFIT [56], LSQ [19], APOT [37], LSQ+ [3], LLSQ [73], DAQ [30], BRECQ [38], EWGS [35], BR [22], OOQ [54], and LLT [68].”
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot (2023, background): “Post-training compression has traditionally been investigated in the context of quantization [23, 34, 29] to reduce the computational cost of quantization-aware training.”
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022, background): “Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, neutral): “Other techniques like Ze-roQuant (Yao et al., 2022), AdaRound (Nagel et al., 2020), and BRECQ (Li et al., 2021) rely on backpropagation to up-date the quantized weights, which may not easily scale up to large model sizes; they also do not outperform GPTQ (Fran-tar et al., 2022), thus not included fo”
Accurate Post Training Quantization With Small Calibration Sets (2021, methodology): “Using BRECQ baseline model we achieved 75.18% top-1 on ResNet50.”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “For example, EasyQuant [46] determines appropriate c min and c max based on training data, and BRECQ [20] introduces Fisher information into the objective.”
Post-Training Quantization on Diffusion Models (2022, neutral): “Therefore, we propose to introduce post-training quantization (PTQ) [21, 3, 16] into DM acceleration.”
SpinQuant: LLM quantization with learned rotations (2024, result): “The results in Table 6 demonstrate that 4-bit quantization yields a ∼ 3 × speedup compared to the 16-bit model.”
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023, methodology): “Other techniques like ZeroQuant [47], AdaRound [28], and BRECQ [23] rely on backpropagation to update the quantized weights, which may not easily scale up to large model sizes; they also do not outperform GPTQ [14], thus not included for study.”
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (2023, neutral): “[LGT21] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu.”
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning (2022, neutral): “1% , which is in line with other post training methods [31, 24].”
QuIP: 2-Bit Quantization of Large Language Models With Guarantees (2023, methodology): “There are many quantization methods which quantize by optimizing this proxy objective [5, 6, 9, 12, 14, 20, 31].”
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (2023, methodology): “While QAT maintains performance by simulating quantization during training, its training cost makes it unsuitable for LLM. PTQ techniques like AdaRound (Nagel et al., 2020) and BRECQ (Li et al., 2021) use gradient optimization to determine optimal rounding, but tuning all weights is time-intensive f”