Fixed point optimization of deep convolutional neural networks for object recognition

By Sajid Anwar, Kyuyeon Hwang, Wonyong Sung. 2015

216 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fixed Point Quantization of Deep Convolutional Networks (2015, methodology): “e speed. Han et al. (2015) and Gong et al. (2014) on the other hand applied codebook based on scalar and vector quantization methods respectively, in order to reduce the model size. In the spirit of (Sajid et al., 2015), we also focus on optimizing DCN models that are pre-trained with ﬂoating point ”
High-Accuracy Low-Precision Training (2018, background): “It has been folklore for many years that neural network inference could be done effectively even with 8-bit arithmetic [31], and there has been much work recently on compressing already-trained networks by (among other things) making some of the weights and activations low-precision [3, 12, 30].”
Training Quantized Nets: A Deeper Understanding (2017, background): “Previous work on obtaining a quantized neural network can be divided into two categories: quantizing pre-trained models with or without retraining [7, 13, 6, 14, 15], and training a quantized model from scratch [4, 5, 3, 1, 16].”
Structured Pruning of Deep Convolutional Neural Networks (2015, methodology): “The fixed-point optimization algorithm is outlined in [16] [17].”
Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets (2015, methodology): “et al. (2014) were able to reduce the precision of gradients to one bit in the training of a neural network using Stochastic Gradient Descent with almost no accuracy loss. Hwang & Sung (2014) and Anwar et al. (2015) quantize the signal (data) and weights in a fully connected network and CNNs and”
Resiliency of Deep Neural Networks under Quantization (2015, background): “In the ﬁxed-point optimization examples shown in Hwang & Sung (2014); Anwar et al. (2015); Shin et al. (2015), neural networks with ternary weights showed quite good performance which was close to that of ﬂoating-point arithmetic.”
Focus: Querying Large Video Datasets with Low Latency and Low Cost (2018, methodology): “oposes various techniques to reduce the running time of CNNs. These techniques include shallow models [21], predicting weights [33], matrix pruning [31,37], model quantization [36], and others (e.g., [20,34,39,41,42,57,62]). Our work is largely orthogonal to these, in that our system is not tied to ”
LCNN: Lookup-Based Convolutional Neural Network (2016, methodology): “Several network quantization methods are proposed by [13, 1, 29, 29, 19].”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “Anwar et al. (2015) quantized the neural network using L2 error minimization and achieved better accuracy on MNIST and CIFAR-10 datasets.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, methodology): “Quantizing a network with L 2 error minimization achieved better accuracy on MNIST and CIFAR-10 datasets in [33].”
Maximizing CNN accelerator efficiency through resource partitioning (2016, methodology): “ the MultiCLP design has 3.3x higher throughput than a Single-CLP design with identical resource budget. Moreover, if ﬁxed-point arithmetic replaces ﬂoating point (as suggested by some recent studies [8]), this greatly increases the number of multipliers and adders available even in existing FPGAs, ”
Early fire detection using convolutional neural networks during surveillance for effective disaster management (2017, background): “Convolutional neural networks have exhibited state-of-the-art performance in a variety of computer vision tasks including image classiﬁcation and retrieval [28–30] , object detection [31,32] , localization [33] , and image segmentation [34] .”
Cognitive computation and communication: A complement solution to cloud for IoT (2016, methodology): “In [48][49], the pre-trained weights of deep CNNs are quantized by ternary representation (+1, 0 and -1) and retrained with 3-bit fixed-point activations.”
Stripes: Bit-serial deep neural network computing (2016, methodology): “This property allows the representation length to be tuned per layer without a loss in accuracy and has been used to simplify hardwired DNN implementations [2] and to reduce memory”
Towards the Limit of Network Quantization (2016, background): “89; Hassibi & Stork, 1993; Han et al., 2015b) and low-precision ﬁxed- or ﬂoating-point implementation of neural networks (Vanhoucke et al., 2011; Hwang & Sung, 2014; Courbariaux et al., 2014; Anwar et al., 2015; Gupta et al., 2015; Lin et al., 2015a) have been also investigatedfor networkcom”
A Survey on Methods and Theories of Quantized Neural Networks (2018, background): “[Anwar et al. , 2015] proposed a layer-wise quantization scheme to reduce the performance degradation.”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, methodology): “Instead of using single-precision 32-bit ﬂoating point (FP32), the calculations and implementation of DL algorithms become much easier in 16-bit ﬂoating-point [112], 8-bit ﬂoating-point [113], and ﬁxed-point representations [114], [115], [116], [117].”
Compact Deep Convolutional Neural Networks With Coarse Pruning (2016, methodology): “The next pruning granularity is removing feature maps Polyak & Wolf (2015); Anwar et al. (2015b).”