A Deep Neural Network Compression Pipeline: Pruning, Quantization, Huffman Encoding

By Song Han, Huizi Mao, William (Bill) J. Dally. 2015

57 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jonathan Frankle
2. Song Han
3. Yi Yang
4. Yang He
5. Qi Tian
6. Yunhe Wang
7. Dacheng Tao
8. Yu Cheng

Citations in the corpus

• Fixed Point Quantization of Deep Convolutional Networks (2015, background): “This can be seen as a process of quantization.”
• Compressing Convolutional Neural Networks in the Frequency Domain (2015, methodology): “[22] further combine the techniques of pruning and quantization to achieve more compression.”
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “A pruning approach (Han et al., 2015b;a) also aims at reducing the total amount of parameters and operations in the entire network.”
• Scalpel: Customizing DNN pruning to the underlying hardware parallelism (2017, unknown): “Deep Compression [19, 20] provides a typical weight pruning technique.”
• The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, methodology): “[4, 3, 2] showed that these techniques could be used to substantially reduce the size of modern image-recognition networks.”
• Distilling Object Detectors with Task Adaptive Regularization (2020, background): “Weight quantization [29,32,9] refers to representing the weights with fewer bits, pruning approach [10,21,27] removes unimportant”
• Compact Deep Convolutional Neural Networks With Coarse Pruning (2016, background): “Network pruning is one promising techique that first learns a function with a suficiently large sized network followed by removing less important connections Yu et al. (2012); Han et al. (2015b); Anwar et al. (2015b).”
• Hardware-oriented Approximation of Convolutional Neural Networks (2016, methodology): “This rounding scheme was successfully used by Gupta et al. (2015) for weight updates of 16-bit fixed point networks.”
• Neuro-Inspired Computing With Emerging Nonvolatile Memorys (2018, methodology): “Recent research efforts from the algorithms perspective have made the attempts, including the network pruning [125] and low-precision (fixed-point) training with stochastic rounding of last few bits [126].”
• In Teacher We Trust: Learning Compressed Models for Pedestrian Detection (2016, background): “When T = 1 , this is the standard softmax function, while higher values of T produce a smoother probability distribution over the classes.”
• Fast, Lean, and Accurate: Modeling Password Guessability Using Neural Networks (2016, methodology): “Other work achieves higher compression ratios for neural networks than we do by using matrix factorization or specialized training methods [51, 96].”
• Pruning Filters for Efficient ConvNets (2016, methodology): “Recently Han et al. (2015; 2016b) report impressive compression rates on AlexNet (Krizhevsky et al. (2012)) and VGGNet (Simonyan & Zisserman (2015)) by pruning weights with small magnitudes and then retraining without hurting the overall accuracy.”