Ultimate tensorization: compressing convolutional and FC layers alike

By Timur Garipov, Dmitrii Podoprikhin, Alexander Novikov, Dmitry P. Vetrov. 2016

206 citations recorded by the source corpus.

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

Subfields: Computational Mathematics, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Variational Dropout Sparsifies Deep Neural Networks (2017, neutral): “One possible approach is to use tensor decompositions (Novikov et al., 2015; Garipov et al., 2016).”
Structured Bayesian Pruning via Log-Normal Multiplicative Noise (2017, background): “tensor decompositions [4, 17], and the other approach is to induce sparsity, e.”
Moonshine: Distilling with Cheap Convolutions (2017, methodology): “Equally, there are many methods for low rank approximations that could be applicable (Sainath et al., 2013; Jaderberg et al., 2014; Garipov et al., 2016).”
A Tensorized Transformer for Language Modeling (2019, background): “This approach was later extended to convolutional [13] and recurrent neural networks [35].”
Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “It is used in Novikov et al. (2015) for compression of fully-connected layers, and Garipov et al. (2016) applies it for convolutional layers.”
T-Basis: a Compact Representation for Neural Networks (2020, methodology): “Based on the conventions set by the prior art (Novikov et al., 2015; Garipov et al., 2016; Wang et al., 2018), we aim to perform reparameterization of existing Neural Network architectures, such as ResNets (He et al., 2016), without altering operator interfaces and data flow.”
Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, methodology): “Table 5 enumerates recent TT compression on CNNs. Note that only Garipov et al. [121] extended tensorizing to Conv.”
Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer (2020, methodology): “Ranks assumed fixed Various types of low-rank decompositions of fully-connected and convolutional layers (with suitable reshaping) have been used [10, 11, 23, 32, 43, 45, 47, 48, 49, 52], as well as tensor decompositions such as CP [11, 30], Tucker [26] and Tensor-Train [14, 38].”
Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions (2018, methodology): “(Garipov et al., 2016) proposed a tensor factorization based method speciﬁcally for compressing convolutional layers.”
Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation (2019, methodology): “In a followup work [4], the same strategy is also applied to convolutional layers.”