Low precision arithmetic for deep learning

By Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David. 2014

175 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Fixed Point Quantization of Deep Convolutional Networks (2015, methodology): “In this section, we will propose an algorithm to convert a ﬂoating point DCN to ﬁxed point.”
Bitwise Neural Networks (2016, background): “It was also shown that 10 bits and 12 bits are enough to represent gradients and storing weights for implementing the state-of-the-art maxout networks even for training the network (Courbariaux et al., 2014).”
Compression of Neural Machine Translation Models via Pruning (2016, methodology): “Several methods involve reducing the precision of the weights or activations (Courbariaux et al., 2015), sometimes in conjunction with specialized hardware (Gupta et al.”
PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Several papers [5, 9] showed that CNNs may be efficiently evaluated using low precision arithmetic, which is important for FPFGA implementations.”
Stripes: Bit-serial deep neural network computing (2016, methodology): “For example most software implementations use 32-bit floating-point [7], [8], [9] while accelerators and some recent GPUs use 16-bit fixed-point [3], [10], [11].”
Deep Learning with Limited Numerical Precision (2015, background): “While preparing this paper, we became aware of a very recent work (Courbariaux et al., 2014) that shares our motivations but adopts an orthogonal approach.”
Channel-Level Acceleration of Deep Face Representations (2015, methodology): “Another family of methods improves hardware utilization [32], [33] employing various techniques such as low-level parallelism, effective memory usage and low precision arithmetic.”
Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets (2015, methodology): “The same is true for the 32-bit precision, as most implementations use only 16 bits (Chen et al. (2014b); Gupta et al. (2015); Courbariaux et al. (2014)).”
Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM (2017, background): “For example, the pretrained weights are quantized into a few bits, usually 4-12 bits, resulting in a minimal loss of performance [25, 10, 4, 24].”
Fault Tolerance in Iterative-Convergent Machine Learning (2018, background): “Training using quantized or low-precision ﬂoating point representations drastically reduces the overhead of computation and communication limitations (Courbariaux et al., 2014; Gupta et al., 2015; Hubara et al., 2017).”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “[20] show that fixed-point arithmetic is sufficient to train deep networks.”
The cascading neural network: building the Internet of Smart Things (2017, result): “A similar result can be obtained when using reduced precision parameters in the network [21,22].”
Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks (2016, background): “However, the training algorithms have a strong tendency to diverge when the precision of network parameters and features are too low (Han et al., 2015; Courbariaux et al., 2014).”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, methodology): “[14] have shown that SGD requires weights with a precision of at least 6 to 8 bits and [16] successfully train DNNs with 12 bits dynamic fixed-point computation.”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “All weights can be stored with low bit precision (Courbariaux et al., 2014; Gupta et al., 2015), edges could be removed (Cires¸an et al., 2011) and HashedNets can be trained on the outputs of larger networks (Ba & Caruana, 2014) — yielding further reductions in memory requirements.”
Efficient Memory Management for GPU-based Deep Learning Systems (2019, methodology): “Half precision (16FP), single bit, and mixed precision numbers have been applied in DNNs [26]–[28].”