Towards the Limit of Network Quantization

By Yoojin Choi, Mostafa El-Khamy, Jungwon Lee. 2016

217 citations recorded by the source corpus.

Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al., 2017), or direct codebook optimization (Rastegari et al., 2016; Hou et al., 2016; Leng et al., 2018; Zhang et al.,…”
Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations (2017, methodology): “2 Adhering to the benchmark of [5, 12, 11], we obtain the compression factor by dividing the bit cost of storing the uncompressed weights as ﬂoats ( 464 , 154 × 32 bits) with the total encoding cost of compressed weights ( i.e. , L × 32 bits for the centers plus the size of the compressed index…”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Some works [74, 248] use k-means on different tensors to determine the quantization steps and levels, while other work [38] applies a Hessian-weighted k-means clustering on weights to minimize the performance loss.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “• DKM requires no additional computation such as Hessian trace (Dong et al., 2020; Choi et al., 2017) or SVD (J.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “The Hessian method was applied to measure the importance of network parameters and therefore improve weight quantization [44].”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “…feasible to ﬁnd q for reasonable model accuracy without a pre-trained model, we assumed q = 2 or q = 4 to estimate speed-up and energy consumptions of major matrix multiplications based on the reports that larger neural networks can be quantized by a smaller number of quantization bits [21], [49].”
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression (2020, methodology): “Moreover, Choi et al. (2016) investigated the drawbacks of conventional quantization methods based on k-means and proposed a Hessian-weighted k-means clustering algorithm as the solution.”
Jointly Sparse Convolutional Neural Networks in Dual Spatial-winograd Domains (2019, unknown): “, in [4–10].”
And the Bit Goes Down: Revisiting the Quantization of Neural Networks (2019, unknown): “This design choice signiﬁcantly reduces their memory footprint but makes structured-based compression algorithms irrelevant.”
Weightless: Lossy Weight Encoding For Deep Neural Network Compression (2017, methodology): “For example, two recent works (Han et al., 2016; Choi et al., 2017) encode pruned and quantized neural networks with sparse matrix representations.”
Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications (2022, unknown): “Similarly to pruning, Hessian-based quantization is shown to be effective in [139].”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, methodology): “We chose the uniform quantizer because of its simplicity and high performance relative to other more sophisticated quantizers such as entropy-constrained k-mean algorithms [34], [35].”
A Survey of Model Compression and Acceleration for Deep Neural Networks (2017, methodology): “It was shown in [11] that Hessian weight could be used to measure the importance of network parameters, and”
A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects (2020, background): “Besides, multivariate quantization is leveraged to represent weights by several values [171], [172].”
Soft-to-Hard Vector Quantization for End-to-End Learned Compression of Images and Neural Networks (2017, methodology): “We adopt the same setting as (Choi et al., 2016) and consider a 32-layer architecture trained for the CIFAR-10 dataset (Krizhevsky & Hinton, 2009).”
Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges (2018, methodology): “It was shown in [11] that Hessian weight could be used to measure the importance of network parameters, and proposed to minimize Hessian-weighted quantization errors in average for clustering network parameters.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, methodology): “While simple and effective, [Choi et al., 2016] pointed out the above quantization method have two drawbacks.”
SBNet: Sparse Blocks Network for Fast Inference (2018, background): “Based on these reasons, most of current approaches to speedup deep neural networks leverage other redundancies such as weight quantization [33, 1], teacher-student network compression [9], etc.”