Recurrent Neural Networks With Limited Numerical Precision

By Joachim Ott, Zhouhan Lin, Y. Zhang, Shih-Chii Liu, Yoshua Bengio. 2016

80 citations recorded by the source corpus.

Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases. This has led to different proposed rounding methods which have been applied so far to only Convolutional Neural Networks and Fully-Connected Networks. This paper addresses the question of how to best reduce weight precision during training in the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets. The results show that the weight binarization methods do not work with the RNNs. However, the stochastic and deterministic ternarization, and pow2-ternarization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datasets therefore providing a path towards training more efficient implementations of RNNs in specialized hardware.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Mixed Precision Training (2017, background): “Another approach to quantize RNNs is proposed in Ott et al. (2016).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “There are a large variety of recurrent models with (2016) tried to quantize the RNNs weight matrices using similar techniques as described in Section 2.”
Model compression via distillation and quantization (2018, background): “…neural networks, e.g. Courbariaux et al. (2015); Rastegari et al. (2016); Hubara et al. (2016); Wu et al. (2016a); Mellempudi et al. (2017); Ott et al. (2016); Zhu et al. (2016), which showed that neural networks can converge to good task solutions even when weights are constrained to…”
TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning (2017, background): “From the perspective of inference acceleration, sparse and quantized Deep Neural Networks (DNNs) have been widely studied, such as [17][18][19][20][21][22][23][24][25].”
Effective Quantization Methods for Recurrent Neural Networks (2016, methodology): “However, in contrast to the extensive study in compression and quantization of convolutional neural networks, little attention has been paid to reducing the computational resource requirements of RNN. (Ott et al., 2016) claims that the weight binarization method does not work with RNNs, and introduc”
Fully Quantized Transformer for Machine Translation (2019, methodology): “Ott et al. (2016) propose an exponential quantization method for RNN weights.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “For example, hysteresis parameters can be introduced to control the level projection by shifting the transition region [151], [159]; pow2-ternarization is also used to ternarize W by introducing the power-of-two quantization for the fractional part [188].”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, unknown): “(2017), pow2-ternaization (Ott et al., 2016), parameter-dependent adaptive threshold (He et al.”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “Recently, some researchers have looked at recurrent neural networks [ Ott et al. , 2016; Hou et al. , 2016; He et al. , 2016b; Clark et al. , 2017; Lu Hou, 2018; Chen Xu and Zha, 2018 ] and other tasks such as semantic segmentation [ Wen et al. , 2016 ] , video processing [ O’Connor and Welling,…”
DRISA: A DRAM-based Reconfigurable In-Situ Accelerator (2017, methodology): “Data quantization in recursive NN (RNN) with 2-bit weights [69] is also an ideal case to run on DRISA.”
A comprehensive survey on model compression and acceleration (2020, background): “The work in Ott et al. (2016) introduces limited numerical precision that can help in reducing weight precision for RNN.”
On-chip training of recurrent neural networks with limited numerical precision (2017, methodology): “[13] have shown that limited numerical precision can be applied in RNNs to reduce computation by quantizing weights and eliminating multiplications in the forward pass.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, unknown): “…cluster (Park, Ahn, and Yoo 2017), trainable quantization (Zhang et al. 2018), ﬁttable quantization scale (Cai et al. 2017), pow2-ternaization (Ott et al. 2016), stochastic weight averaging (Shin, Boo, and Sung 2020), increasing the size of the neural network (Ka-pur, Mishra, and Marr 2017),…”