Training deep neural networks with low precision multiplications

By Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David. 2014

607 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…to optimize accelerator memory utilization and beneﬁt faster computation, including meth-ods like gradient checkpointing and low precision training (Courbariaux et al., 2014; Ott et al., 2018), memory efﬁcient optimizers (Shazeer and Stern, 2018; Gupta et al., 2018) and frameworks supporting…”
Scaling Neural Machine Translation (2018, methodology): “Previous research considered training and inference with reduced numerical precision for neural networks (Simard and Graf, 1993; Courbariaux et al., 2015; Sa et al., 2018).”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “…different bit-widths have been studied: 1-bit methods (Courbariaux and Bengio, 2016; Rastegari et al., 2016; Courbariaux et al., 2015), 2 to 3-bit (Zhu et al., 2017; Choi et al., 2019), 4-bits (Li et al., 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et al., 2019).”
8-bit Optimizers via Block-wise Quantization (2021, methodology): “, 2019), more bits (Courbariaux et al., 2014), or a variable amount of bits (Gong et al.”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Until recently, the use of extremely low-precision networks (binary in the extreme case) was believed to substantially degrade the network performance (Courbariaux et al., 2014).”
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017, background): “Another emerging approach is low bit networks [4, 22, 11].”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “The performance of highly quantized networks ( e.g., binarized) were believed to be very poor due to the destructive property of binary quantization [35].”
Deep Learning with Limited Numerical Precision (2015, background): “Secondly, modern 5While preparing this paper, we became aware of a very recent work (Courbariaux et al., 2014) that shares our motivations but adopts an orthogonal approach.”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “All weights can be stored with low bit precision (Courbariaux et al., 2014; Gupta et al., 2015), edges could be removed (Cireşan et al., 2011) and HashedNets can be trained on the outputs of larger networks (Ba & Caruana, 2014) — yielding further reductions in memory requirements.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, background): “Recently, fixed-point based approaches are explored in [5, 25].”
Fixed Point Quantization of Deep Convolutional Networks (2015, background): “Fixed point implementation of DCNs has been explored in earlier works (Courbariaux et al., 2014; Gupta et al., 2015).”
Binarized Neural Networks (2016, background): “Until recently, the use of extremely lowprecision networks (binary in the extreme case) was believed to be highly destructive to the network performance (Courbariaux et al., 2014). Soudry et al. (2014) and Cheng et al.”
Bayesian Compression for Deep Learning (2017, background): “Several studies show that this has little to no effect on network accuracy when using 16bit weights [Merolla et al., 2016, Gupta et al., 2015, Courbariaux et al., 2014, Venkatesh et al., 2016, Chai et al., 2017].”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, background): “Recent research efforts (Courbariaux et al., 2014; Kim & Smaragdis, 2016; Rastegari et al., 2016; Li & Liu, 2016; Merolla et al., 2016) have considerably reduced both model size and computation complexity by using low bitwidth weights and low bitwidth activations.”
Soft Weight-Sharing for Neural Network Compression (2017, background): “A number of papers have appeared that encode the weights of a neural network with limited precision (say 8 bits per weight), effectively cashing in on this ”bits-back” argument (Gupta et al., 2015; Courbariaux et al., 2014; Venkatesh et al., 2016).”
NISP: Pruning Networks Using Neuron Importance Score Propagation (2017, neutral): “Other studies focused on fixed point computation rather than exploiting the CNN redundancy [4, 29].”