Weighted-Entropy-Based Quantization for Deep Neural Networks

By Eunhyeok Park, Junwhan Ahn, Sungjoo Yoo. 2017

272 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

The case for 4-bit precision: k-bit Inference Scaling Laws (2022, background): “…centering (Krishnamoorthi, 2018; Jacob et al., 2017), learned data types that are found through clustering (Gong et al., 2014; Han et al., 2015; Choi et al., 2016; Park et al., 2017), or direct codebook optimization (Rastegari et al., 2016; Hou et al., 2016; Leng et al., 2018; Zhang et al., 2018).”
Secure Evaluation of Quantized Neural Networks (2019, methodology): “this extensive research body have enabled modern quantized neural networks to have essentially the same accuracy as their full-precision counterparts [12, 68, 26, 30, 52], even with large CNN architectures like AlexNet [43], VGGNet [59], GoogleNet [60] and ResNet [31].”
SqueezeNext: Hardware-Aware Neural Network Design (2018, background): “Therefore, most works in this area have to perform some form of retraining to recover accuracy [8, 9, 20, 15, 21].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “One could also consider using other metrics such as entropy [184], although MSE is the most common method used.”
Knowledge distillation: A good teacher is patient and consistent (2021, background): “Weights quantization [25, 17, 31, 45] and decomposition [5, 23, 43, 9] aim to accelerate and reduce the memory footprint of CNNs by replacing large matrices operations with their lightweight approximations.”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Recent approaches quantize both weights and activations through using uniform quantization that uses identical bit-width for all layers [67, 8, 39], or mixed precision quantization that uses different bit-widths for different layers or even channels [51, 4, 57].”
POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging (2022, background): “2017), and others have proposed techniques to quan-tize activations while performing full-precision multiply-accumulates (MACs).”
A Survey on Methods and Theories of Quantized Neural Networks (2018, methodology): “In [Park et al. , 2017], the authors adopted a way which is similar to vector quantization.”
Empirical Analysis of Knowledge Distillation Technique for Optimization of Quantized Deep Neural Networks (2019, background): “, 2015), weight cluster (Park et al., 2017), trainable quantization (Zhang et al.”
Constraint-Aware Deep Neural Network Compression (2018, background): “The results show that, in a fair comparison with the same hardware and software platforms, our method obtains compara- ble latency performance to optimized GSL, while requiring no hardware-specific tuning and providing generality to other constraint types besides latency.”
Accurate and Efficient 2-bit Quantized Neural Networks (2019, methodology): “Training Complexity Quantization at inference Enlarge Network size WEQ (Park et al., 2017) Weight clustering (iterative) Non-uniform N BQ (Zhou et al., 2016) Histogram equalization (iterative) Non-uniform N HWGQ (Cai et al., 2017) Lloyd’s algorithm (iterative) Non-uniform N LQ-Nets (Zhang et al., 20”
A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (2018, methodology): “quirements of DNNs, with minor eect on the overall accuracy. These model compression techniques include weight pruning [9,10,11,12,13,14,15,16], sparsity regularization [17,18,19], weight clustering [9,20,21], and low rank approximation [22,23], etc. A simple but eective weight pruning method has be”
SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks (2018, methodology): “Activations quantization has also been investigated with frameworks created for varying activation bitwidths [33] and both weights and activations [23].”
PACT: Parameterized Clipping Activation for Quantized Neural Networks (2018, methodology): “For comparisons, we include accuracy results reported in the following prior work: DoReFa (Zhou et al. (2016)), BalancedQ (Zhou et al.”
Learning to Quantize Deep Networks by Optimizing Quantization Intervals With Task Loss (2018, unknown): “- 61.0 - 68.4 - 66.7 - - - - - BalancedQ [28] - - - 59.4 - - - - - - - 55.7 TSQy[25] - - - - - - - - - - - 58.0 SYQy[6] - - - - - - - - - - - 55.8 Zhuang et al. [30] - - - - - - - - - 58.1 - 52.5 WEQ [20] - - - - - - - - - 55.9 54.9 50.6 Table 2. The top-1 accuracy (%) of low bit-width networks on R”
DKM: Differentiable K-Means Clustering Layer for Neural Network Compression (2021, background): “Figure 3: Multi-dimensional DKM can increase the weight entropy (ew) with a fixed compression ratio target (cr) by increasing a dimension (d), which may improve the model accuracy (Park et al., 2017).”
C-GOOD: C-code Generation Framework for Optimized On-device Deep Learning (2018, methodology): “Approximate computing can achieve almost the same accuracy with significantly reduced resource requirements and is used in various approximation techniques such as pruning[22], quantization[24], and low-rank approximation[18].”