Relaxed Quantization for Discretized Neural Networks

By Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, Max Welling. 2018

149 citations recorded by the source corpus.

Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “Louizos et al. (2019) uses a Gumbel-softmax (Jang, Gu, and Poole 2017) but requires 2 hyper-parameters and has no bitwidth tuning.”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, methodology): “Although many methods exist that do quantization-aware training (Jacob et al., 2018; Louizos et al., 2019) and get excellent results, these methods require a user to spend signiﬁcant time on re-training models and hyperparameter tuning.”
Mixed Precision DNNs: All you need is a good parametrization (2019, background): “As proposed in (Jang et al., 2016; Maddison et al., 2016; Louizos et al., 2019), the categorical distribution can be relaxed to a concrete distribution – a smoothed approximation of the categorical distribution – such that the ELBO becomes differentiable under reparametrization.”
LSQ+: Improving low-bit quantization through learnable offsets and better initialization (2020, methodology): “Simulated quantization-aware training methods and improvements for these are discussed in [9, 12, 17].”
Data-Free Quantization Through Weight Equalization and Bias Correction (2019, methodology): “Some higher level approaches [16, 21] report slightly higher results than our baseline model, likely due to a better training procedure than used in the standard Pytorch Resnet18 model.”
Bayesian Bits: Unifying Quantization and Pruning (2020, methodology): “…32/32 69.68 100 (Jacob et al., 2018) 8/8 70.38 6.25 TWN (Li et al., 2017a) 2/32 61.80 5.95 LR-Net (Shayer et al., 2017)* 1/32 59.90 4.58 XNOR-net (Rastegari et al., 2016) 1/1 51.20 0.14 RQ (Louizos et al., 2019) 5/5 65.10 2.54 PACT* (Choi et al., 2018) 4/4 69.20 3.12 PACT* (Choi et al., 2018) 2”
Differentiable Joint Pruning and Quantization for Hardware Efficiency (2020, methodology): “ith powerof-two bit-restricted quantization as presented in Section 3.5. bit-width Test Acc. MACs(G) BOPs(G) BOP comp. ratio Baseline 32/32 93.0% 0.613 629 – TWN [17] 2/32 92.56% 0.613 39.23 16.03 RQ [19] 8/8 93.30% 0.613 39.23 16.03 RQ [19] 4/4 92.04% 0.613 9.81 64.12 WAGE [32] 2/8 93.22% 0.613 9.8”