Deep Learning with Limited Numerical Precision

By Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan. 2015

2,190 citations recorded by the source corpus.

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Efficiency & Compression

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, background): “Many of those previous studies [19, 20, 43, 27] however mostly focus on CNN models with relatively few layers.”
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019, background): “A separate line of study leverages quantization to derive smaller models (Gupta et al. [2015]).”
Mixed Precision Training (2017, methodology): “Gupta et al. (2015) demonstrate that 16 bit ﬁxed point representation can be used to train CNNs on MNIST and CIFAR-10 datasets without accuracy loss.”
In-datacenter performance analysis of a tensor processing unit (2017, background): “Relative GPU and TPU includes host server overhead.”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “To reduce memory usage, we freeze the first 80% of the weights (64/80 transformer layers) to the pretrained values, share parameters between policy and value functions, and train with reduced precision using bfloat16 as in Rae et al. (2021) and stochastic rounding (Gupta et al., 2015).”
Red Teaming Language Models with Language Models (2022, methodology): “To reduce memory usage, we freeze the first 80% of the weights (64/80 transformer layers) to the pretrained values, share parameters between policy and value functions, and train with reduced precision using bfloat16 and stochastic rounding (Gupta et al., 2015).”
The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning (2017, background): “While we do not yet understand why these interpolated classifiers generalize so well to unseen data, there is ample empirical evidence for their excellent generalization performance in deep neural networks (Gupta et al., 2015; Chaudhari et al., 2016; Zhang et al., 2016; Huang et al., 2016; Sagun et ”
Measuring the Algorithmic Efficiency of Neural Networks (2020, methodology): “We don’t account for gains from being able to use lower precision computation [Gupta et al., 2015].”
Teaching language models to support answers with verified quotes (2022, methodology): “To fit the model in TPU memory, we shard the model over 128 TPU v3 cores, rematerialize activations every 3 transformer blocks, freeze the embedding layers and train in low precision using bfloat16 and stochastic rounding (Gupta et al., 2015).”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “[14] and [15] show that randomized or stochastic rounding can be used to provide unbiased discretization.”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “16-bit fixed-point representation (Gupta et al., 2015) for a compression factor of 14 over double-precision floating point).”
Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization (2019, background): “Limiting the number of bits in the floating point representation is studied at [8, 13, 25].”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “However, for the task of training, there are few works that perform all operations in fixed-point [20, 23, 33, 39, 57].”
Communication trade-offs for synchronized distributed SGD with large step size (2019, background): “Some of the work in this direction can be found in Zhang et al. (2017); Wen et al. (2017); Wangni et al. (2017); Sa et al. (2015); Na et al. (2017); Gupta et al. (2015); Alistarh et al. (2016); Khirirat et al. (2018).”
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware (2018, methodology): “We thus quantize all inputs and weights of a DNN so as to operate over integers [20], and embed these integers in a field Zp of integers modulo a prime p (p is larger than all values computed during a DNN evaluation).”
cpSGD: Communication-efficient and differentially-private distributed SGD (2018, background): “To address this issue, several recent works have focused on reducing the communication cost of distributed learning algorithms via gradient quantization and sparsification [32, 17, 33, 20, 21, 4, 34].”
Local SGD Converges Fast and Communicates Little (2018, background): “For instance, by limiting the number of bits in the floating point representation [12, 24, 30], or random quantization [4, 40].”
QSGD: Communication-Optimal Stochastic Gradient Descent, with Applications to Training Neural Networks (2016, background): “Proposals such as [37, 14, 18] investigate deterministic and stochastic quantization for weights, activations, and gradients during training, with as little as one bit per dimension.”