8-Bit Approximations for Parallelism in Deep Learning

By Tim Dettmers. 2015

203 citations recorded by the source corpus.

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.

Subfields: Artificial Intelligence, Computational Mechanics, Efficiency & Compression, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Efficiency & Compression

In Optimization & Training

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Finally, we use dynamic exponent quantization (Dettmers, 2016), which uses an indicator bit to separate an exponent bit region and a linear quantization region.”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “Dynamic Tree quantization (Dettmers, 2016) is a method that yields low quantization error for both small and large magnitude values.”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “Quantization is commonly performed by way of reducing floating-point dynamic range [56, 64, 88].”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “Continuing with the model and data parallelism approach introduced by Krizhevsky (2014), Dettmers (2016) recently built a GPU cluster consisting of 96 units and showed that by compressing gradients and nonlinear activations into 8-bit representations, speed-ups of up to 50 times were achievable.”
A Study of BFLOAT16 for Deep Learning Training (2019, background): “Literature shows that various different reduced precision data representations have been investigated which can be broadly classified into two types: the more standard floating-point based formats [26, 13, 10] and custom fixed point based formats [34, 6, 16, 20, 21].”
Ternary Neural Networks with Fine-Grained Quantization (2017, background): “Many of the recent reduced precision work, look at the low precision only for the weights while retaining the activations in full precision [23, 21, 6, 15, 16, 14, 4].”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, background): “There are some other techniques that have been proposed in last few years [271,272,273].”
Stochastic Controlled Averaging for Federated Learning with Communication Compression (2023, background): “Notable examples include Sign-SGD (Seide et al., 2014; Bernstein et al., 2018), low-bit fixed rounding (Dettmers, 2016), Q-SGD (Alistarh et al., 2017), Turn-Grad (Wen et al., 2017), and natural compression (Horv´oth et al., 2022).”
Rethinking floating point for deep learning (2018, background): “[4] shows a kind of 8 bit ﬂoating point for communicating gradients, but this is not used for general computation.”
Neural Networks and Deep Learning (2018, unknown): “Furthermore, algorithmic tricks like using 8-bit approximations of the gradients [98] can help in speeding up the”