Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks

By Darryl Dexu Lin, Sachin S. Talathi. 2016

48 citations recorded by the source corpus.

It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the well-accepted solutions facilitating the training of low precision fixed point networks is stochastic rounding. However, to the best of our knowledge, the source of the instability in training neural networks with noisy gradient updates has not been well investigated. This work is an attempt to draw a theoretical connection between low numerical precision and training algorithm stability. In doing so, we will also propose and verify through experiments methods that are able to improve the training performance of deep convolutional networks in fixed point.

Subfields: Artificial Intelligence, Computational Mechanics, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Christos Louizos
3. Max Welling
4. Yoshua Bengio
5. Nuno Vasconcelos
6. Mohit Bansal
7. Luke S. Zettlemoyer
8. Xiaodong He

In Computational Mechanics

1. Yunhui Guo
2. Zhaowei Cai
3. Xiaodong He
4. V. Sreekanth Annapureddy
5. Max Welling
6. Nuno Vasconcelos
7. Erik J. Bekkers
8. David W. Romero

In Statistical and Nonlinear Physics

1. Yunhui Guo
2. Max Welling
3. Xiaodong He
4. Nuno Vasconcelos
5. Christos Louizos
6. Jun Zhu
7. Jiaxin Shi
8. Feng Zhu

Citations in the corpus

• Fixed Point Quantization of Deep Convolutional Networks (2015, background): “L G ] 2 J un 2016 the second approach may produce networks with superior accuracy numbers (Rastegari et al., 2016; Lin & Talathi, 2016), it requires tight integration between the network de-sign, training and implementation, which is not always feasible.”
• Bayesian Compression for Deep Learning (2017, background): “these to be relevant scenarios because (i) can be applied with already existing frameworks such as Tensorflow [1], (ii) is a practical scheme given upcoming GPUs and frameworks will be designed to work with low and mixed precision arithmetics [43, 23].”
• Deep Learning with Low Precision by Half-Wave Gaussian Quantization (2017, background): “This is called the “gradient mismatch” problem [23].”
• OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks (2022, background): “However, these techniques can compromise the accuracy of the neural networks, make training unstable, and require careful implementation to be deployed successfully [59, 51].”
• A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “• The quantized activations can lead to “gradient mismatch” problem [Lin and Talathi, 2016] which means that there is a discrepancy between the quantized activation with the computed backward gradient.”