Low precision storage for deep learning

By Matthieu Courbariaux, Yoshua Bengio, Jean-Pierre David. 2014

51 citations recorded by the source corpus.

We train a set of state of the art neural networks, the Maxout networks (Goodfellow et al., 2013a), on three benchmark datasets: the MNIST, CIFAR10 and SVHN, with three distinct storing formats: floating point, fixed point and dynamic fixed point. For each of those datasets and for each of those formats, we assess the impact of the precision of the storage on the final error of the training. We find that very low precision storage is sufficient not just for running trained networks but also for training them. For example, Maxout networks state-of-the-art results are nearly maintained with 10 bits for storing activations and gradients, and 12 bits for storing parameters.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “The compression rates of all methods could be further improved by learning and storing parameters in lower precision [9, 21].”
PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory (2016, unknown): “The data format we use is dynamic fixed point [69].”
Learning compact recurrent neural networks (2016, methodology): “Other techniques include induc ing zeros in the parameter matrices via sparsity regularizers [ 6]; toring weights in low fixed-precision formats [7, 8]; using specific param-”
A high performance FPGA-based accelerator for large-scale convolutional neural networks (2016, methodology): “In addition, as references [9, 22, 23] have shown that low precision quantization is sufficient for CNN implementations,16-bit fixed point data are adopted for the input data, weights and the intermediate data.”
Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks (2016, background): “However, the training algorithms have a strong tendency to diverge when the precision of network parameters and features are too low (Han et al., 2015; Courbariaux et al., 2014).”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “The obvious solution is to fit the neural network within budget by reducing the number of nodes n, n in layers `, `+ 1 or by reducing the bit precision of the weight matrices (Courbariaux et al., 2014).”
Structured Transforms for Small-Footprint Deep Learning (2015, methodology): “Other techniques include freezing M to be a random matrix as motivated via approximations to kernel functions [20], storing M in low fixed-precision formats [7, 24], using specific parameter sharing mechanisms [3], or training smaller models on outputs of larger models (“distillation”) [11].”
NVIDIA Tensor Core Programmability, Performance & Precision (2018, background): “While several studies have shown that deep neural networks are tolerant to low precision calculation [13], [14], [15], such studies are still in their infancies in HPC.”
Hardware-oriented Approximation of Convolutional Neural Networks (2016, methodology): “Dynamic ﬁxed point (Williamson, 1991; Courbariaux et al., 2014) is a solution to this problem.”