Compression of Deep Neural Networks on the Fly

By Guillaume Soulié, Vincent Gripon, Maëlys Robert. 2015

26 citations recorded by the source corpus.

Thanks to their state-of-the-art performance, deep neural networks are increasingly used for object recognition. To achieve the best results, they use millions of parameters to be trained. However, when targetting embedded applications the size of these models becomes problematic. As a consequence, their usage on smartphones or other resource limited devices is prohibited. In this paper we introduce a novel compression method for deep neural networks that is performed during the learning phase. It consists in adding an extra regularization term to the cost function of fully-connected layers. We combine this method with Product Quantization (PQ) of the trained weights for higher savings in storage consumption. We evaluate our method on two data sets (MNIST and CIFAR10), on which we achieve significantly larger compression rates than state-of-the-art methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Compressing Convolutional Neural Networks in the Frequency Domain (2015, background): “A few recent works [52, 61] also adopt the quantization techniques.”
More is Less: A More Complicated Network with Less Inference Complexity (2017, critical): “Generally, most of these sparsity-based methods [22, 25, 31] integrate the sparsity property as a regularizer into the learning of parameters, which usually harms the performance of network.”
Deep Model Compression: Distilling Knowledge from Noisy Teachers (2016, methodology): “Soulie et al. [28] used a regularization technique instead to coarsely quantize the weights of fully connected layers.”