Differentiable Model Compression via Pseudo Quantization Noise
By Alexandre D'efossez, Yossi Adi, Gabriel Synnaeve. 2021
64 citations recorded by the source corpus.
We propose DiffQ a differentiable method for model compression for quantizing model parameters without gradient approximations (e.g., Straight Through Estimator). We suggest adding independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. DiffQ is differentiable both with respect to the unquantized weights and the number of bits used. Given a single hyper-parameter balancing between the quantized model size and accuracy, DiffQ optimizes the number of bits used per individual weight or groups of weights, in end-to-end training. We experimentally verify that our method is competitive with STE based quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation. For instance, on the ImageNet dataset, DiffQ compresses a 12 layers transformer-based model by more than a factor of 8, (lower than 4 bits precision per weight on average), with a loss of 0.3% in model accuracy. Code is available at github.com/facebookresearch/diffq.
Subfields: Artificial Intelligence, Efficiency & Compression, Optimization & Training, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Armand Joulin
- Herv'e J'egou
- Edouard Grave
- Piotr Bojanowski
- Matthieu Cord
- Gabriel Synnaeve
- Luke S. Zettlemoyer
- Jianfeng Gao
In Efficiency & Compression
- Armand Joulin
- Gabriel Synnaeve
- Edouard Grave
- Herv'e J'egou
- Hugo Touvron
- Matthieu Cord
- Piotr Bojanowski
- Alaaeldin El-Nouby
In Optimization & Training
- Matthieu Cord
- Herv'e J'egou
- Hugo Touvron
- Alaaeldin El-Nouby
- Gautier Izacard
- Jakob Verbeek
- Mathilde Caron
- Sein Park
In Signal Processing
- Gabriel Synnaeve
- Gautier Izacard
- Herv'e J'egou
- Jakob Verbeek
- Mathilde Caron
- Hugo Touvron
- Eunhyeok Park
- Armand Joulin
Citations in the corpus
- ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, background): “This suggests that they may be compatible with parameter pruning, or better, with modern quantization techniques that induce sparsity at training time, such as Quant-Noise [17] and DiffQ [16].”
- Overcoming Oscillations in Quantization-Aware Training (2022, unknown): “This phenomenon was recently also observed by D´efossez et al. (2021).”
- NIPQ: Noise Injection Pseudo Quantization for Automated DNN Optimization (2022, methodology): “The last line of related study is DiffQ [26], which is based on PQN-based QAT for weight quantization.”