On-chip training of recurrent neural networks with limited numerical precision

By Taesik Na, Jong Hwan Ko, Jaeha Kung, Saibal Mukhopadhyay. 2017

42 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Peter Richtárik
2. Sebastian U. Stich
3. Martin Jaggi
4. Konstantin Mishchenko
5. Samuel Horváth
6. Dmitry Kovalev
7. Aymeric Dieuleveut
8. Mehrdad Mahdavi

In Computer Vision and Pattern Recognition

1. Martin Jaggi
2. Peter Richtárik
3. Sebastian U. Stich
4. Jean-Baptiste Cordonnier
5. Aymeric Dieuleveut
6. Zhouhan Lin
7. Shih-Chii Liu
8. Samuel Horváth

In Electrical and Electronic Engineering

1. Sebastian U. Stich
2. Martin Jaggi
3. Peter Richtárik
4. Mehrdad Mahdavi
5. Farzin Haddadpour
6. Konstantin Mishchenko
7. Mohammad Mahdi Kamani
8. Aymeric Dieuleveut

In Efficiency & Compression

1. Sebastian U. Stich
2. Martin Jaggi
3. Peter Richtárik
4. Farzin Haddadpour
5. Mehrdad Mahdavi
6. Mohammad Mahdi Kamani
7. Aymeric Dieuleveut
8. Samuel Horváth

In Optimization & Training

1. Peter Richtárik
2. Sebastian U. Stich
3. Martin Jaggi
4. Mehrdad Mahdavi
5. Konstantin Mishchenko
6. Farzin Haddadpour
7. Dmitry Kovalev
8. Kumar Kshitij Patel

Citations in the corpus

• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Limiting the number of bits in the floating point representation is studied at [8, 13, 25].”
• Natural Compression for Distributed Deep Learning (2019, background): “In their most basic form, these schemes decrease the number of bits used to represent floating point numbers forming the communicated d-dimensional vectors [17, 27, 33], thus reducing the size of the communicated message by a constant factor.”
• Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the work in this direction can be found in Zhang et al. (2017); Wen et al. (2017); Wangni et al. (2017); Sa et al. (2015); Na et al. (2017); Gupta et al. (2015); Alistarh et al. (2016); Khirirat et al. (2018).”
• Local SGD Converges Fast and Communicates Little (2018, background): “For instance, by limiting the number of bits in the floating point representation [12, 24, 30], or random quantization [4, 40].”
• Sparsified SGD with Memory (2018, background): “Several papers consider approaches that limit the number of bits to represent floating point numbers [13, 24, 31].”
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction (2019, background): “In the simplest form, these schemes limit the number of bits that are used to represent floating point numbers (Gupta et al., 2015; Na et al., 2017), reducing the size of a d -dimensional (gradient) vector by a constant factor.”
• Recurrent Neural Networks With Limited Numerical Precision (2016, methodology): “In [15] we additionally provide a r X i v : 1611 .”