Hardware-oriented Approximation of Convolutional Neural Networks

By Philipp Gysel, Mohammad Motamedi, Soheil Ghiasi. 2016

327 citations recorded by the source corpus.

High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

Citations in the corpus

Model compression via distillation and quantization (2018, background): “(2016); Gysel et al. (2016); Mishra et al. (2017), which combine quantization, weight sharing, and careful coding of network weights, to reduce the size of state-of-the-art deep models by orders of magnitude, while at the same time speeding up inference.”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “Performing the backward pass with ﬂoating point is important, as accumulating the gradients in quantized precision can result in zero-8 gradient or gradients that have high error, especially in low-precision [42, 79, 80, 106, 156, 181, 198, 223].”
MnnFast: A Fast and Scalable System Architecture for Memory-Augmented Neural Networks (2019, unknown): “Quantization decreases overall bitwidths [28, 45, 55].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Instead of using single-precision 32-bit ﬂoating point (FP32), the calculations and implementation of DL algorithms become much easier in 16-bit ﬂoating-point [112], 8-bit ﬂoating-point [113], and ﬁxed-point representations [114], [115], [116], [117].”
HarDNet: A Low Memory Traffic Network (2019, background): “While there are only a few papers addressing lossless compression of feature maps, lossy compression of feature maps has been intensively studied in research of model precision manipulation and approximation [8, 11, 14, 28, 29].”
Hierarchical Compression of Deep Convolutional Neural Networks on Large Scale Visual Recognition for Mobile Applications (2016, methodology): “At final step, we quantize the floating point compressed DCNN from previous steps, we analyze the dynamic range of the input/output activations and the parameters for a given layer as a similar method in [6].”
YodaNN: An Architecture for Ultralow Power Binary-Weight CNN Acceleration (2016, methodology): “For LeNet and Cifar-10 the additional error introduced by this quantization is less than 0.3% and 2%, respectively, even when the word width has been constrained to 4-bit [21].”
Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017, background): “Furthermore, when weights are quantized to powers of two, the multiplication can be replaced with a bitshift [122, 135].”
Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks? (2017, methodology): “Many researchers have shown (e.g., [6,7,10,11]) that it is possible to represent data in much less than 32-bits, demonstrating the use of 8-4 bits (depending on the network) leads to only a small reduction in accuracy compared to full precision.”
Gist: Efficient Data Encoding for Deep Neural Network Training (2018, background): “Second, in line with previous research [15], [8], [16], we represent all the data structures throughout the network in FP16 format and then train the network (shown as All-FP16 ).”
Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks (2016, background): “More recently, several works have touched upon the issue of training deep networks with low numerical precision (Gupta et al., 2015; Lin et al., 2015; Gysel et al., 2016).”
Distilling Object Detectors With Fine-Grained Feature Imitation (2019, methodology): “Previous works mainly focus on Quantization [12, 14, 36, 28] which efficiently reduces computation and model size, and network pruning [14, 13, 1, 34] that prunes redundant connections in large models.”
A Practical Convolutional Neural Network as Loop Filter for Intra Frame (2018, methodology): “Since CPU and GPU do not support DFPs, they are simulated by float points similar to [10].”
YodaNN: An Ultra-Low Power Convolutional Neural Network Accelerator Based on Binary Weights (2016, methodology): “Sev-1 eral reduced-precision implementations have been proposed recently, relying on 16-bit, 12-bit or 10-bit of accuracy for both operands and weights [10], [11], [12], [13], [14], exploiting the intrinsic resiliency of CNNs to quantization and approximation [15], [16].”