Effective Quantization Methods for Recurrent Neural Networks

By Qinyao He, He Wen, Shuchang Zhou, Yuxin Wu, Cong Yao, Xinyu Zhou, Yuheng Zou. 2016

82 citations recorded by the source corpus.

Reducing bit-widths of weights, activations, and gradients of a Neural Network can shrink its storage size and memory usage, and also allow for faster training and inference by exploiting bitwise operations. However, previous attempts for quantization of RNNs show considerable performance degradation when using low bit-width weights and activations. In this paper, we propose methods to quantize the structure of gates and interlinks in LSTM and GRU cells. In addition, we propose balanced quantization methods for weights to further reduce performance degradation. Experiments on PTB and IMDB datasets confirm effectiveness of our methods as performances of our models match or surpass the previous state-of-the-art of quantized RNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Mixed Precision Training (2017, methodology): “He et al. (2016c) propose quantized variants of the GRU (Cho et al., 2014) and Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) cells to use fewer bits for weights and activations.”
Model compression via distillation and quantization (2018, methodology): “Neural networks are extremely effective for solving several real world problems, like image classification (Krizhevsky et al., 2012; He et al., 2016a), translation (Vaswani et al., 2017), voice synthesis (Oord et al., 2016) or reinforcement learning (Mnih et al., 2013; Silver et al., 2016).”
Knowledge Distillation from Internal Representations (2019, methodology): “Other model compression techniques include quantization (Hubara et al. 2017; He et al. 2016; Courbariaux et al. 2016) and weights pruning (Han, Mao, and Dally 2015).”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “…(Zhang et al. 2018), fittable quantization scale (Cai et al. 2017), pow2ternaization (Ott et al. 2016), parameter-dependent adaptive threshold (He et al. 2016b), increasing the size of the neural network (Kapur, Mishra, and Marr 2017), or quantization interval learning (Jung et al. 2019).”
$L1$ -Norm Batch Normalization for Efficient Training of Deep Neural Networks (2018, unknown): “Although a lot of quantization methods have been proposed to reduce the memory cost and accelerate the computation [20], [21], [22], the BN layer still remains in ﬂoat32 precision or avoided for simplicity [23].”
A comprehensive survey on model compression and acceleration (2020, methodology): “The experimental results on ResNet, WideResNet and VGG16 architecture with CIFAR10, CIFAR100, and ImageNet dataset show that the proposed factor transfer method outperforms classical knowledge distillation and attention transfer (Zagoruyko and Komodakis 2017) methods.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “For example, AlexNet [Krizhevsky et al. , 2012] requires 200MB memory, VGG-Net [ Simonyan and Zisserman, 2014 ] requires 500MB memory and ResNet-101 [He et al. , 2016a] requires 200MB memory.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Usually, the straight-through estimator (STE) is used to circumvent this problem [147], [152], [153].”
Compression of Acoustic Event Detection Models With Quantized Distillation (2019, methodology): “Quantization methods have been studied from the perspective of different model architectures [13, 14] as well the specific applications (e.”
Fully Quantized Transformer for Machine Translation (2019, methodology): “Meanwhile, He et al. (2016) propose modiﬁca-tions to the gates and interlinks of quantized LSTM and GRU (Cho et al., 2014) cells, as well as a balanced quantization method for weights.”
Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “2 We measure the amount of compression of the LSTM/GRU/FastRNN layer of the network We conducted a study to test whether KPRNNs are compatible with quantization using the approach in (He et al., 2016b).”