And the Bit Goes Down: Revisiting the Quantization of Neural Networks

By Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, Herv'e J'egou. 2019

158 citations recorded by the source corpus.

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the network outputs rather than its weights. The principle of our approach is that it minimizes the loss reconstruction error for in-domain inputs. Our method only requires a set of unlabelled data at quantization time and allows for efficient inference on CPU by using byte-aligned codebooks to store the compressed weights. We validate our approach by quantizing a high performing ResNet-50 model to a memory size of 5MB (20x compression factor) while preserving a top-1 accuracy of 76.1% on ImageNet object classification and by compressing a Mask R-CNN with a 26x factor.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, methodology): “To get a more compressed model, model quantization is also a useful technique, which has been widely explored in some CNN-based models (Stock et al., 2020; Polino et al., 2018).”
Differentiable Model Compression via Pseudo Quantization Noise (2021, background): “We also compare to PQ (Stock et al. 2019), as reported by Fan et al.”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Other quantization methods such as Vector Quantization (VQ) and PQ [36] quantize blocks of weights simulatneously to achieve higher compression rate [8, 37, 38, 39].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “While many works [6, 7, 8, 9, 10] aim at reducing the memory footprint of classifiers and feature extractors, inference speed is equally important, with high throughput corresponding to better energy efficiency.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Product quantization [74, 219, 248] is an extension of vector quantization, where the weight matrix is divided into submatrices and vector quantization is applied to each submatrix.”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “Although combining quantization and distillation has been explored in convolutional neural networks (CNNs) (Polino et al., 2018; Stock et al., 2020; Kim et al., 2019), using knowledge distillation to train quantized BERT has not been studied.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “For example, when combined with [19], [5] showed good compression vs.”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, background): “And, many related studies have been explored in convolutional neural networks (CNNs) models Polino et al. (2018), Stock et al. (2019), Kim et al. (2019).”
nuQmm: Quantized MatMul for Efficient Inference of Large-Scale Generative Language Models (2022, background): “…feasible to ﬁnd q for reasonable model accuracy without a pre-trained model, we assumed q = 2 or q = 4 to estimate speed-up and energy consumptions of major matrix multiplications based on the reports that larger neural networks can be quantized by a smaller number of quantization bits [21], [49].”
Continuous-in-Depth Neural Networks (2020, background): “Moreover, any ResNet is generally difficult to compress without significantly decreasing model accuracy [74, 12, 69].”
Training with Quantization Noise for Extreme Fixed-Point Compression (2020, methodology): “Other quantization methods such as Vector Quantization (VQ) and PQ [36] quantize blocks of weights simulatneously to achieve higher compression rate [8, 37, 38, 39].”
SPINN: synergistic progressive inference of neural networks over device and cloud (2020, methodology): “ut significant impact on the accuracy of the subsequent classifiers (i.e. <0.65 percentage point drop across all exits of the examined CNNs). Our scheme differs from both i) weights-only reduction [17, 67, 86], which minimises the model size rather than activations’ size and ii) all-layers quanti”