Tensorizing Neural Networks
By Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, Dmitry P. Vetrov. 2015
998 citations recorded by the source corpus.
Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train [17] format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks [21] we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.
Subfields: Artificial Intelligence, Computational Mathematics, Hardware and Architecture
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Mohit Bansal
- Dacheng Tao
- Sergey Levine
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Percy Liang
In Computational Mathematics
- Timothy M. Hospedales
- Yongxin Yang
- Andrzej S. Cichocki
- Dmitry P. Vetrov
- Qibin Zhao
- Anh Huy Phan
- M. A. Carreira-Perpiñán
- Yerlan Idelbayev
In Hardware and Architecture
- M. A. Carreira-Perpiñán
- Yerlan Idelbayev
- Edouard Grave
- Dmitry P. Vetrov
- Anima Anandkumar
- Armand Joulin
- Timothy M. Hospedales
- Victor S. Lempitsky
Citations in the corpus
- Recent advances in convolutional neural networks (2015, methodology): “[163] generalize the low-rank ideas, where they treat the weight matrix as multi-dimensional tensor and apply a Tensor-Train decomposition [164] to reduce the number of parameters of the fully-connected layers.”
- Fast ConvNets Using Group-Wise Brain Damage (2015, methodology): “Our work focuses on the task of speeding up convolutional layers (as they represent the speed bottleneck) and is therefore complimentary to approaches that focus on the reduction of size/memory footprint of fully-connected layers [11, 18, 38, 42, 46].”
- Variational Dropout Sparsifies Deep Neural Networks (2017, neutral): “One possible approach is to use tensor decompositions (Novikov et al., 2015; Garipov et al., 2016).”
- LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks (2018, methodology): “Representative techniques include quantization [6, 34, 29, 54, 52, 21, 39, 53, 22, 3, 55, 9, 18], pruning [13, 12, 17, 36], low-rank decomposition [7, 8, 24, 27, 38, 51, 49], hashing [4], and deliberate architecture design [23, 19, 50].”
- Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications (2015, methodology): “More recently, various methods based on vector quantization (Gong et al., 2014), hashing techniques (Chen et al., 2015), circulant projection (Cheng et al., 2015), and tensor train decomposition (Novikov et al., 2015) were proposed and showed better compression capability than SVD.”
- SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, background): “…structured simplification such as pruning filters (Li et al. (2017)), structured sparsity with regular-izers (Wen et al. (2016)), low-rank approximation (Jaderberg et al. (2014)), matrix and tensor factorization (Novikov et al. (2015)), and sparsification using expander graphs (Prabhu et al. (2018)).”
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise (2017, background): “tensor decompositions [4, 17], and the other approach is to induce sparsity, e.”
- PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions (2015, background): “Several papers have demonstrated that it is possible to compress the parameters of the fully-connected layers (where most CNN parameters reside) with a marginal error increase [4, 21, 27].”
- Deep Multi-task Representation Learning: A Tensor Factorisation Approach (2016, methodology): “In deep learning, tensor factorisation has been used to exploit factorised tensors’ fewer parameters than the original (e.g., 4-way convolutional kernel) tensor, and thus compress and/or speed up the model, e.g., [Lebedev et al., 2015, Novikov et al., 2015].”
- Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges (2022, background): “Both EMNIST and Fashion-MNIST are considered easier datasets, as there are many works that achieve high training accuracy after few epochs of training [169, 170].”
- A Tensorized Transformer for Language Modeling (2019, methodology): “Tensorizing Neural Networks [24] have combined the idea of reshaping weights of fully-connected layers into high-dimensional tensors and representing them in Tensor Train format [25].”
- Convolutional Neural Networks using Logarithmic Data Representation (2016, background): “Interesting directions point towards matrix factorization (Denton et al., 2014) and tensorification (Novikov et al., 2015) by leveraging structure of the fully-connected (FC) layers.”
- Doubly Convolutional Neural Networks (2016, methodology): “[19] use a multi-linear transform (Tensor-Train decomposition) to attain reduction of the number of parameters in the linear layers of CNNs.”
- Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification (2017, methodology): “In recent years, many compression methods have been proposed, including knowledge distillation[1, 9, 20], weight quantization[5, 18], weight pruning[7, 25] and weight decomposition[3, 17].”
- Separable Layers Enable Structured Efficient Linear Substitutions (2019, methodology): “A number of methods have reduced stored size or computational cost in neural networks by providing efficient alternatives to fully connected layers; these include HashedNet (Chen et al., 2015), TensorTrain (Novikov et al., 2015) and ACDC (Moczulski et al., 2016).”
- Ultimate tensorization: compressing convolutional and FC layers alike (2016, methodology): “By combing this convolutional approach with the work [1] for fully-connected layers, we compressed a convolutional network 80 × times.”
- Pruning neural networks without any data by iteratively conserving synaptic flow (2020, background): “While there are a variety of approaches to compressing neural networks, such as novel design of micro-architectures [15, 16, 17], dimensionality reduction of network parameters [18, 19], and training of dynamic sparse networks [20, 21, 22], in this work we will focus on neural network pruning.”
- Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “It is used in Novikov et al. (2015) for compression of fully-connected layers, and Garipov et al. (2016) applies it for convolutional layers.”
- Exploring Interpretable LSTM Neural Networks over Multi-Variable Data (2019, background): “Do et al. (2017); Novikov et al. (2015) proposed to represent hidden states as matrices.”
- In Teacher We Trust: Learning Compressed Models for Pedestrian Detection (2016, background): “…function L used for training the student is a combination of the soft loss L soft , the cross-entropy loss be-tween the soft outputs of the student and teacher, as well as the hard loss L hard , the standard classification cross-entropy loss between the student outputs and the ground truth labels.”