Compressing Deep Convolutional Networks using Vector Quantization

By Yunchao Gong, Liu Liu, Ming Yang, Lubomir D. Bourdev. 2014

1,233 citations recorded by the source corpus.

Deep convolutional neural networks (CNN) has become the most promising method for object recognition, repeatedly demonstrating record breaking results for image classification and object detection in recent years. However, a very deep CNN generally involves many layers with millions of parameters, making the storage of the network model to be extremely large. This prohibits the usage of deep CNNs on resource limited hardware, especially cell phones or other embedded devices. In this paper, we tackle this model storage issue by investigating information theoretical vector quantization methods for compressing the parameters of CNNs. In particular, we have found in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods. Simply applying k-means clustering to the weights or conducting product quantization can lead to a very good balance between model size and recognition accuracy. For the 1000-category classification task in the ImageNet challenge, we are able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

FitNets: Hints for Thin Deep Nets (2014, background): “Other works related to quantization schemes (Chen et al., 2010; J´egou et al., 2011; Gong et al., 2014) aim at reducing storage requirements.”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “The most commonly used techniques include quantization (Gong et al., 2014), weights pruning (Han et al.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “, 2020) and quantization (Gong et al., 2014; Zafrir et al., 2019; Stock et al., 2021) as well as early exit strategies for inference (Liu et al.”
FastText.zip: Compressing text classification models (2016, methodology): “Some use vector quantization (Gong et al., 2014) while others binarize the network (Courbariaux et al., 2016).”
Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some methods approximate model weights using quantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximation and tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “For example, there are quantization methods where a data type is optimized with additional input data (Rastegari et al., 2016; Frantar et al., 2022) or from the weights of the model alone (Gong et al., 2014).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “Gong et al. (2014) compressed deep convnets using vector quantization, which resulteds in only a 1% accuracy loss.”
Learning both Weights and Connections for Efficient Neural Network (2015, background): “While these large neural networks are very powerful, their size consumes considerable storage, memory bandwidth, and computational resources.”
Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding (2015, methodology): “Deep Fried Convnets(Yang et al., 2014) worked on fully connected layers and reduced the parameters by less than 4 × .”
Patient Knowledge Distillation for BERT Model Compression (2019, background): “Another direction is weight quantization (Gong et al., 2014; Polino et al., 2018), in which connection weights are constrained to a set of discrete values, allowing weights to be represented by fewer bits.”
Quantized Convolutional Neural Networks for Mobile Devices (2015, methodology): “These approaches mostly focus on compressing parameters in fully-connected layers [11, 2], and none of them can provide acceleration for the test-phase computation.”
Secure Evaluation of Quantized Neural Networks (2019, background): “4 This is a long-standing research area, with initial works already dating back to the 1990s [5, 26, 49, 62], and this extensive research body have enabled modern quantized neural networks to have essentially the same accuracy as their full-precision counterparts [14, 29, 32, 53, 65].”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “ch cases were found in our experiments but can be an interesting area to explore in future. The network is pruned with simple methods like L1 norm and more compression tricks like vector quantization [17] and group sparsity regularization [1] can be applied. In our work, we show that reducing the nu”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “r of bits required to represent each parameter from 32-bit ﬂoats to 8 bits or fewer. Different quantization techniques such as ﬁxed-point quantization (Vanhoucke et al., 2011) or vector quantization (Gong et al., 2014) achieve different compression ratios and accuracies but also require different so”
Fixed Point Quantization of Deep Convolutional Networks (2015, methodology): “This can be seen as a process of quantization.”
Recent advances in convolutional neural networks (2015, neutral): “[84] investigate the information theoretical vector quantization methods for compressing the parameters of CNNs, and they obtain parameter prediction results similar to those”
AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks (2020, neutral): “Beyond scalar quantization, vector quantization was widely adopted too in model compression for parameter sharing (Gong et al., 2014; Wu et al., 2018).”
AMC: AutoML for Model Compression and Acceleration on Mobile Devices (2018, background): “Tensor factorization [30, 15, 27, 35] decomposes weights into light-weight pieces, for example [51, 11, 14] proposed to accelerate the fully connected layers with truncated SVD; Jaderberg et al .”
Data-free Parameter Pruning for Deep Neural Networks (2015, methodology): “Our method can be applied on top of most networks with a fully connected layer to give a smaller network.”