Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets

By Patrick Judd, Jorge Albericio, Tayler H. Hetherington, Tor M. Aamodt, Natalie D. Enright Jerger, Raquel Urtasun, Andreas Moshovos. 2015

103 citations recorded by the source corpus.

This work investigates how using reduced precision data in Convolutional Neural Networks (CNNs) affects network accuracy during classification. More specifically, this study considers networks where each layer may use different precision data. Our key result is the observation that the tolerance of CNNs to reduced precision data not only varies across networks, a well established observation, but also within networks. Tuning precision per layer is appealing as it could enable energy and performance improvements. In this paper we study how error tolerance across layers varies and propose a method for finding a low precision configuration for a network while maintaining high accuracy. A diverse set of CNNs is analyzed showing that compared to a conventional implementation using a 32-bit floating-point representation for all layers, and with less than 1% loss in relative accuracy, the data footprint required by these networks can be reduced by an average of 74% and up to 92%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, background): “, by rounding) after its training with floating point activations is usually not sufficient (Judd et al., 2015).”
vDNN: Virtualized deep neural networks for scalable, memory-efficient neural network design (2016, background): “Another redundancy mitigating approach uses quantization [48] or reduced precision [49] to reduce the number of bits required to model the network.”
Binarized Neural Networks (2016, methodology): “Other research (Judd et al., 2015; Gong et al., 2014) aimed to compress a fully trained high precision network by using a quantization or matrix factorization method.”
Stripes: Bit-serial deep neural network computing (2016, background): “When considering the characteristics of neural networks [1],”
When Neurons Fail - Technical Report (2016, background): “This has been recently demonstrated experimentally [13] without however any theoretical explanation until the present paper.”