Deep Learning with Low Precision by Half-Wave Gaussian Quantization

By Zhaowei Cai, Xiaodong He, Nuno Vasconcelos. Computer Vision and Pattern Recognition, 2017

537 citations recorded by the source corpus.

The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the widely used ReLU non-linearity is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/28b79a6a752433958fa4d7ac82ac304b7087400a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Towards Accurate Post-training Network Quantization via Bit-Split and Stitching (2020, methodology): “Note that for each convolutional kernel q , we introduce a ﬂoating-point scale factor α like in previous low-bit quantization methods (Lebedev et al., 2014; Hu et al., 2018b; Cai et al., 2017).”
MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization (2019, methodology): “In the methods HWGQ [3] and Bi-real [12], ∂Q(r)∂r is manually defined, but they focused on activation quantization.”
Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks (2017, methodology): “Another category of algorithms that aim at network acceleration is weight and activation quantization (Rastegari et al. 2016; Cai et al. 2017; Chen et al. 2015), but they usually suffer from an evident drop in performance despite yielding signiﬁcant speed-up.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “More recently, [Cai et al. , 2017] proposed an half-wave Gaussian quantizer to approximate the ReLU unit.”
Fully Quantized Network for Object Detection (2019, background): “ory usage [33, 11, 30], quantizing both network parameters and activations to accelerate specific types of layers in the network using bitwise operations [35, 31, 14, 4], and quantizing network gradients to speed up distributed training [2].”
Learned Step Size Quantization (2019, methodology): “…al. (2016), later work focused on ﬁtting the quantizer to the data, either based on statistics of the data distribution (Li & Liu, 2016; Zhou et al., 2016; Cai et al., 2017; McKinstry et al., 2018) or seeking to minimize quantization error during training (Choi et al., 2018c; Zhang et al., 2018).”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, unknown): “For instance, many non-uniform quantization methods have been designed for bell-shaped distributions of the weights and activations that often involve long tails [12, 25, 60, 113, 143, 173].”
Mixed Precision Training With 8-bit Floating Point (2019, background): “There have been numerous attempts [13],[27],[6],[25],[4] to train deep neural networks at lower precision (< 16-bits) with varying degrees of success.”
Overcoming Oscillations in Quantization-Aware Training (2022, methodology): “Earlier work in QAT (Cai et al., 2017; Yin et al., 2019) investigated different variations of the STE and demonstrate experimentally that the derivative of the clipped ReLU is a better approximation than the vanilla STE (derivative of identity).”
Joint Training of Low-Precision Neural Network with Quantization Interval Parameters (2018, methodology): “Ours (λ = 0) PACT [10] DoReFa [9] HWGQ [20] BalancedQ [27]”
PACT: Parameterized Clipping Activation for Quantized Neural Networks (2018, methodology): “One of the earliest studies in weight quantization schemes (Hwang & Sung (2014) and Courbariaux et al. (2015)) show that it is indeed possible to quantize weights to 1-bit (binary) or 2-bits (ternary), enabling an entire DNN model to fit effectively in resource-constrained platforms (e.g., mobile de”
SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks (2018, methodology): “Improving the network learnability under low-precision weights and activations was analysed in [2].”
F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization (2022, background): “There is also binarization network, which only applies 1-bit (Rastegari et al., 2016; Hubara et al., 2016; Cai et al., 2017; Bulat et al., 2020; Guo et al., 2021).”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Cai et al. [160] maintained the binarization of W while extending the A quantization to multiple bits in the positive region and approximated the quantization function via modiﬁed ReLU to help the gradient-based BP. Li et al. [181] and Deng et al. [151] extrapolated the quantization of both W and A…”