Training with Quantization Noise for Extreme Model Compression

By Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herv'e J'egou, Armand Joulin. 2020

267 citations recorded by the source corpus.

We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods where the approximations introduced by STE are severe, such as Product Quantization. Our proposal is to only quantize a different random subset of weights during each forward, allowing for unbiased gradients to flow through the other weights. Controlling the amount of noise and its form allows for extreme compression rates while maintaining the performance of the original model. As a result we establish new state-of-the-art compromises between accuracy and model size both in natural language processing and image classification. For example, applying our method to state-of-the-art Transformer and ConvNet architectures, we can achieve 82.5% accuracy on MNLI by compressing RoBERTa to 14MB and 80.0 top-1 accuracy on ImageNet by compressing an EfficientNet-B3 to 3.3MB.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2020) and quantization (Gong et al., 2014; Zafrir et al., 2019; Stock et al., 2021) as well as early exit strategies for inference (Liu et al.”
Training data-efficient image transformers & distillation through attention (2020, methodology): “We have employed stochastic depth [29], which facilitates the convergence of transformers, especially deep ones [16, 17].”
Stable and low-precision training for large-scale vision-language models (2023, background): “One major direction is to quantize convolutional neural networks often for fast and memory efficient usage on edge devices [78, 6, 13, 79, 23, 77, 30].”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “, 2020), product quantization with quantization noise (Fan et al., 2020), TernaryBERT (Zhang et al.”
Linformer: Self-Attention with Linear Complexity (2020, methodology): “This technique can be further improved through Quantization Aware Training (Jacob et al., 2018; Fan et al., 2020), where the weights are quantized during training and the gradients are approximated with the Straight-Through Estimator.”
Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “In a similar spirit, Fan et al. (2020) sample for each layer and iteration whether to use the quantized or unquantized weight.”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “We believe this can be addressed separately by model distillation and quantization (Polino et al., 2018; Fan et al., 2020).”
Going deeper with Image Transformers (2021, methodology): “This method is already popular in NLP [21, 22] to train deeper architectures.”
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021, methodology): “Some examples of transformer-based model quantization with alternative quantization schemes include Fan et al. (2020); Darvish Rouhani et al. (2020).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “While STE is the mainstream approach [218, 280], other approaches have also been explored in the literature [2, 25, 31, 59, 142, 160].”
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020, methodology): “, 2020), post-training quantization GOBO (Zadeh and Moshovos, 2020), as well as Quant-Noise which uses product quantization (Fan et al., 2020); and (ii) Other Compression Methods: including weight-sharing method ALBERT (Lan et al.”
Movement Pruning: Adaptive Sparsity by Fine-Tuning (2020, methodology): “Quantization has been applied to a variety of modern large architectures [Fan et al., 2020b, Zafrir et al., 2019, Gong et al., 2014] providing high memory compression rates at the cost of no or little performance.”
BinaryBERT: Pushing the Limit of BERT Quantization (2020, methodology): “Existing attempts try to quan-tize pre-trained models (Zafrir et al., 2019; Shen et al., 2020; Fan et al., 2020) to even as low as ternary values (2-bit) with minor performance drop (Zhang et al., 2020).”
DynaBERT: Dynamic BERT with Adaptive Width and Depth (2020, background): “There have been some attempts to compress and accelerate inference of the Transformer-based models using low-rank approximation [15, 12], weight-sharing [4, 12], knowledge distillation [21, 23, 10, 28], quantization [1, 33, 22, 8] and pruning [16, 17, 3, 25].”
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization (2021, methodology): “…(2019), Fan et al. (2019); dynamic network training Liu et al. (2019), Hou et al. (2020) and quantization Zafrir et al. (2019), Shen et al. (2020), Fan et al. (2020), Krishnamoorthi (2018), etc. Compared with other compression methods, quantization is widely used to compress neural network models…”
Learned Token Pruning for Transformers (2021, background): “These can be broadly categorized as follows: (i) efficient architecture design [8, 21, 24, 26, 27, 36, 45, 47, 50, 60]; (ii) knowledge distillation [23, 38, 42–44]; (iii) quantization [2, 4, 13, 15, 25, 40, 58, 59, 61]; and (iv) pruning.”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, methodology): “For example, when combined with [19], [5] showed good compression vs.”
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023, unknown): “While several recent works have explored language model compression, they are mostly focused on smaller models (Zafrir et al., 2019; Fan et al., 2020; Shen et al., 2020b; Zadeh et al., 2020; Bai et al., 2021; Qin et al., 2021; Liu et al., 2022b) like BERT (Devlin et al., 2019) or BART (Lewis et…”
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference (2020, background): “Recent work has also explored quantization for contextualized word representations, generally showing that quantization-aware training is necessary to achieve reasonable end task performance (Zafrir et al., 2019; Fan et al., 2020).”