Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression

By Jiahao Su, Jingling Li, Bobby Bhattacharjee, Furong Huang. 2018

23 citations recorded by the source corpus.

We propose tensorial neural networks (TNNs), a generalization of existing neural networks by extending tensor operations on low order operands to those on high order ones. The problem of parameter learning is challenging, as it corresponds to hierarchical nonlinear tensor decomposition. We propose to solve the learning problem using stochastic gradient descent by deriving nontrivial backpropagation rules in generalized tensor algebra we introduce. Our proposed TNNs has three advantages over existing neural networks: (1) TNNs naturally apply to high order input object and thus preserve the multi-dimensional structure in the input, as there is no need to flatten the data. (2) TNNs interpret designs of existing neural network architectures. (3) Mapping a neural network to TNNs with the same expressive power results in a TNN of fewer parameters. TNN based compression of neural network improves existing low-rank approximation based compression methods as TNNs exploit two other types of invariant structures, periodicity and modulation, in addition to the low rankness. Experiments on LeNet-5 (MNIST), ResNet-32 (CIFAR10) and ResNet-50 (ImageNet) demonstrate that our TNN based compression outperforms (5% test accuracy improvement universally on CIFAR10) the state-of-the-art low-rank approximation based compression methods under the same compression rate, besides achieving orders of magnitude faster convergence rates due to the efficiency of TNNs.

Subfields: Computational Mathematics, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mathematics

1. Simon Lacoste-Julien
2. Yuan Xie
3. Song Han
4. Saurabh Pitre
5. By Lei Deng
6. Max Welling
7. Luc Van Gool
8. Guoqi Li

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Song Han
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Andrew Zisserman
6. Trevor Darrell
7. Cordelia Schmid
8. Dengxin Dai

In Statistical and Nonlinear Physics

1. Simon Lacoste-Julien
2. Song Han
3. Max Welling
4. Luc Van Gool
5. Guoqi Li
6. Stamatios Georgoulis
7. Markus Nagel
8. Tijmen Blankevoort

In Model Architectures

1. Luc Van Gool
2. Simon Lacoste-Julien
3. Yuan Xie
4. Song Han
5. Saurabh Pitre
6. By Lei Deng
7. Max Welling
8. Guoqi Li

In Efficiency & Compression

1. Tijmen Blankevoort
2. Markus Nagel
3. Max Welling
4. Song Han
5. Luc Van Gool
6. Yuan Xie
7. Guoqi Li
8. Luping P. Shi

Citations in the corpus

• Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “Some methods approximate model weights using quantization and hashing [Gong et al., 2014, Courbariaux et al., 2015], or low-rank approximation and tensor factorization [Denil et al., 2013, Lebedev et al., 2015, Su et al., 2018].”
• T-Basis: a Compact Representation for Neural Networks (2020, methodology): “Tensor decomposition Similarly to filter decomposition, various low-rank parameterizations can be utilized for weight tensors of a convolutional neural network (Su et al., 2018).”
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks (2019, methodology): “An extensive study of applying CP-decomposition, Tucker decomposition, and tensor-train decomposition in combination with factorizing kernel dimensions were published by Su et al. (2018).”
• Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey (2020, background): “Therefore, most research studies pay attention to compress FC layers, whereas the Conv layers remain unchanged [4], [118]–[120].”