A Survey on Methods and Theories of Quantized Neural Networks

By Yunhui Guo. 2018

275 citations recorded by the source corpus.

Deep neural networks are the state-of-the-art methods for many real-world tasks, such as computer vision, natural language processing and speech recognition. For all its popularity, deep neural networks are also criticized for consuming a lot of memory and draining battery life of devices during training and inference. This makes it hard to deploy these models on mobile or embedded devices which have tight resource constraints. Quantization is recognized as one of the most effective approaches to satisfy the extreme memory requirements that deep neural network models demand. Instead of adopting 32-bit floating point format to represent weights, quantized representations store weights using more compact formats such as integers or even binary numbers. Despite a possible degradation in predictive performance, quantization provides a potential solution to greatly reduce the model size and the energy consumption. In this survey, we give a thorough review of different aspects of quantized neural networks. Current challenges and trends of quantized neural networks are also discussed.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Secure Evaluation of Quantized Neural Networks (2019, background): “For a broader survey of the various of the different quantization techniques that exist we direct the reader to Guo [29].”
Up or Down? Adaptive Rounding for Post-Training Quantization (2020, methodology): “One effective way to optimize neural networks for inference is neural network quantization (Krishnamoorthi, 2018; Guo, 2018).”
A Review on Deep Learning in UAV Remote Sensing (2021, methodology): “The survey conducted in [Guo, 2018] gives an important overview of quantization methods.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Compared with recent surveys on accelerators without compression [14], a certain category of compression approaches [15], or a rapid overview of multiple categories on only the algorithm side [16], [17], we present a more systematic summary on most existing methodologies, show more detailed…”
And the Bit Goes Down: Revisiting the Quantization of Neural Networks (2019, background): “As there exists a large body of literature for network compression, we review the work closest to ours and refer the reader to the two recent surveys [18, 19] for a more comprehensive overview.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Network quantization [123, 81] involves replacing datatypes with reduced width types, for example, replacing 32-bit floating point (FP32) with 8-bit integers (INT8).”
Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (2020, methodology): “atural language processing (NLP) tasks. This has resulted in an explosion of interest around techniques to reduce the memory footprint and energy consumption of neural network training and inference (Guo, 2018). Although there are a number of methods to address some of these issues for inference, th”
A Systematic Review on Model Watermarking for Neural Networks (2020, background): “• Rounding [Guo, 2018]: reduce the precision of the parameters.”
Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks (2022, background): “In the same direction, quantization [13] should be dissected to know how a given regularization algorithm influences either training a quantized neural network and performing the quantization after training.”