Learning low-precision neural networks without Straight-Through Estimator(STE)

By Zhi Gang Liu, Matthew Mattina. 2019

43 citations recorded by the source corpus.

The Straight-Through Estimator (STE) is widely used for back-propagating gradients through the quantization function, but the STE technique lacks a complete theoretical understanding. We propose an alternative methodology called alpha-blending (AB), which quantizes neural networks to low precision using stochastic gradient descent (SGD). Our AB method avoids STE approximation by replacing the quantized weight in the loss function by an affine combination of the quantized weight w_q and the corresponding full-precision weight w with non-trainable scalar coefficient alpha and (1- alpha). During training, alpha is gradually increased from 0 to 1; the gradient updates to the weights are through the full precision term, (1-alpha) * w, of the affine combination; the model is converted from full-precision to low precision progressively. To evaluate the AB method, a 1-bit BinaryNet on CIFAR10 dataset and 8-bits, 4-bits MobileNet v1, ResNet_50 v1/2 on ImageNet are trained using the alpha-blending approach, and the evaluation indicates that AB improves top-1 accuracy by 0.9\%, 0.82\% and 2.93\% respectively compared to the results of STE based quantization.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Ocean Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yossi Adi
2. Gabriel Synnaeve
3. Alexandre D'efossez
4. Yuan Xie
5. Guoqi Li
6. Luping P. Shi
7. By Lei Deng
8. Song Han

In Computer Vision and Pattern Recognition

1. Yuan Xie
2. Guoqi Li
3. Luping P. Shi
4. Gabriel Synnaeve
5. By Lei Deng
6. Song Han
7. Yossi Adi
8. Alexandre D'efossez

In Ocean Engineering

1. Alexandre D'efossez
2. Gabriel Synnaeve
3. Yossi Adi

Citations in the corpus

• Differentiable Model Compression via Pseudo Quantization Noise (2021, methodology): “The limitations of STE methods for quantization were first noticed by Liu and Mattina (2019).”
• A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Other notable research includes using pulse training to approximate the derivative of discontinuous points [45], or replacing the quantized weights with an affine combination of floating point and quantized parameters [161].”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Using pulse train to approximate the derivative of discontinuous points [151], designing proper STE formats [154], or gradually rather than immediately closing the gradient BP path to the high-precision data [155] also works.”
• Mixed Precision DNNs: All you need is a good parametrization (2019, background): “Quantized DNNs apply quantizers Q : R → { q 1 , ..., q I } to discretize the weights and/or activations of a DNN (Han et al., 2015; Zhou et al., 2017; Li et al., 2016; Liu & Mattina, 2019; Cardinaux et al., 2018; Jain et al., 2019; Bai et al., 2018).”
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, unknown): “Alpha-Blending (AB) [152] was proposed as a replacement for STE.”