Convolutional Neural Networks using Logarithmic Data Representation

By Daisuke Miyashita, Edward H. Lee, Boris Murmann. 2016

461 citations recorded by the source corpus.

Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “However, in contrast to to Zhou et al., we did not ﬁnd it useful to initialize the network with weights obtained by training the network with full precision weights.”
QUOTIENT: Two-Party Secure Neural Network Training and Prediction (2019, background): “However, for the task of training, there are few works that perform all operations in fixed-point [20, 23, 33, 39, 57].”
LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “[37] used logarithmic quantization and improve the inference efficiency via the bitshift operation.”
HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision (2019, background): “To address the ﬁrst issue, different quantization schemes such as uniformly discretizing logarithmic-domain have been proposed [24].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Note that the calibration process is often conducted in parallel with the ﬁnetuning process for QAT. use a logarithmic distribution [175, 274], where the quantization steps and levels increase exponentially instead of linearly.”
Training Quantized Nets: A Deeper Understanding (2017, methodology): “Experiments show that networks using 16-bit fixed-point representations with stochastic rounding can deliver results nearly identical to 32-bit floating-point computations [4], while lowering the precision down to 3-bit fixed-point often results in a significant performance degradation [18].”
Training and Inference with Integers in Deep Neural Networks (2018, methodology): “However, non-linear quantization method like logarithmic representation (Miyashita et al., 2016; Zhou et al., 2017)”
Towards Accurate Post-training Network Quantization via Bit-Split and Stitching (2020, background): “, 2015), data representation (Miyashita et al., 2016), multi-bit quantization (Lin et al.”
Rethinking floating point for deep learning (2018, methodology): “The approximation log 2 (1 + x ) ≈ x for x ∈ [0 , 1] could also be used [24], but this adds signiﬁcant error, especially with repeated sums. σ ± ( x ) need only be evaluated if one wishes to keep the partial sum in the log domain.”
Scalable Methods for 8-bit Training of Neural Networks (2018, background): “While several works [6, 5] have shown that training at 16-bit is sufficient for most networks, more aggressive quantization schemes were also suggested [24, 16, 14, 10].”
Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017, background): “Furthermore, when weights are quantized to powers of two, the multiplication can be replaced with a bitshift [122, 135].”
Cognitive computation and communication: A complement solution to cloud for IoT (2016, methodology): “Another work which implements log-based coding on data representation with non-uniformly distributed values reduces bit precision to 3-bits and eliminates expensive multipliers for simple bit shifts [37].”
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks (2019, background): “Besides, there are other non-uniform methods like [30, 13, 3, 35, 37, 38, 44] which may need delicate hardware to get acceleration.”
Ternary Neural Networks with Fine-Grained Quantization (2017, methodology): “Many of the recent reduced precision work, look at the low precision only for the weights while retaining the activations in full precision [23, 21, 6, 15, 16, 14, 4].”
ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning (2017, methodology): “Another related work is the uniform quantization strategy in log domain (Miyashita et al., 2016), which is similar to our approach when the data distribution is “log uniform.””
The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning (2016, methodology): “Another related work is the uniform quantization strategy in log domain Miyashita et al. (2016), which is similar to our approach when the data distribution is “log uniform.””
WRPN: Wide Reduced-Precision Networks (2017, background): “Due to such efﬁciency beneﬁts, there are many existing works which propose low-precision deep neural networks (DNNs) [25, 13, 15, 7, 22], even down to 2-bit ternary mode [27, 12, 23] and 1-bit binary mode [26, 4, 17, 5, 21].”
Ultra-Low Precision 4-bit Training of Deep Neural Networks (2020, methodology): “[29] proposed a training scheme based on pure logarithmic 5-bit representations and demonstrated good results on the AlexNet model (on CIFAR10).”
HarDNet: A Low Memory Traffic Network (2019, background): “While there are only a few papers addressing lossless compression of feature maps, lossy compression of feature maps has been intensively studied in research of model precision manipulation and approximation [8, 11, 14, 28, 29].”
Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy (2017, methodology): “Quantization using low-precision numerics (Vanhoucke et al., 2011; Zhou et al., 2016; Lin et al., 2015; Miyashita et al., 2016; Gupta et al., 2015; Zhu et al., 2016; Rastegari et al., 2016; Courbariaux et al., 2015; Umuroglu et al., 2016; Mishra et al., 2017) and model compression (Buciluǎ et al.”