Improving Neural Network Quantization without Retraining using Outlier Channel Splitting

By Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Christopher De Sa, Zhiru Zhang. 2019

351 citations recorded by the source corpus.

Quantization can improve the execution latency and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs, while this work examines the less-studied topic of quantizing a floating-point model without (re)training. DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS), which duplicates channels containing outliers, then halves the channel values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Multiple different metrics have been proposed to ﬁnd the best range, including minimizing Mean Squared Error (MSE) between original unquantized weight distribution and the corresponding quantized values [40, 214, 221, 273].”
Q-Diffusion: Quantizing Diffusion Models (2023, methodology): “This work explores the quantization [50, 8, 49, 7, 29] of the noise estimation model used in the diffusion model to accelerate the denoising of all time steps.”
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks (2019, background): “Note that slipshod clipping of outliers can jeopardize the network a lot (Zhao et al., 2019) although they may only take 1-2% of all weights in one layer.”
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time (2023, methodology): “There are three main techniques: quantization (Han et al., 2015; Jacob et al., 2018; Nagel et al., 2019; Zhao et al., 2019), pruning or sparsity (Molchanov et al., 2016; Liu et al., 2018; Hoefler et al., 2021), and distillation (Hinton et al., 2015; Tang et al., 2019; Touvron et al., 2021).”
AutoScale: Energy Efficiency Optimization for Stochastic Edge Inference Using Reinforcement Learning (2020, methodology): “The frameworks and middleware also enable efﬁcient inference at the edge— various NN optimizations, such as quantization [15], [30], [52], [62], [65], [111], [119], weight compression [43], [68], and graph pruning [112], [118] can be employed before the DNNs are deployed.”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “Post-training quantization methods [2, 29, 6] optimize neural networks for quantization without full training and using a little amount of data.”
A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (2020, background): “Among inference quantization approaches, post training quantization usually does not require access to the partial/full training dataset, and it does not need to re-train/fine-tune the quantized model [1, 2, 3, 4, 5].”
Accurate Post Training Quantization With Small Calibration Sets (2021, methodology): “…combined with our methods, such as bias correction (Banner et al., 2018), equalization (Meller et al., 2019), and outlier channel splitting (Zhao et al., 2019), we did not ﬁnd it necessary: our results demonstrate that our relatively simple pipeline yields state of the art accuracy on both…”
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models (2023, neutral): “These algorithms can be broadly categorized into three groups: ( 1 ) quantization (Han et al., 2015; Jacob et al., 2018; Nagel et al., 2019; Zhao et al., 2019), which involves mapping model parameters or activations from high-precision data types to low-precision counterparts, such as using 8-bit…”
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming (2020, methodology): “…combined with our methods, such as bias correction (Banner et al., 2018), equalization (Meller et al., 2019), and outlier channel splitting (Zhao et al., 2019), we did not ﬁnd it necessary: our results demonstrate that our relatively simple pipeline yields state of the art accuracy on both…”
Post-Training Quantization for Vision Transformer (2021, methodology): “Most of the existing post-training quantization methods are designed for convolutional neural networks [3, 21, 30] or recurrent neural networks [36].”
AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance (2020, methodology): “To fully execute DNN inference at the edge, many optimizations, such as model architecture search [80, 87, 101], quantization [13,26,43,53,55,92,97], weight compression [15, 37, 58, 61] and graph pruning [93, 96], have been proposed.”
ZeroQ: A Novel Zero Shot Quantization Framework (2020, methodology): “In addition, [44] proposes an outlier channel splitting (OCS) method to solve the outlier channel problem.”
Loss aware post-training quantization (2019, background): “Another efficient approach quantization is to change the distribution of the tensors, making it more suitable for quantization, such as equalizing the weight ranges (Nagel et al. 2019) or splitting outlier channels (Zhao et al. 2019).”