Compression of fully-connected layer in neural network by Kronecker product
By Jia-Nan Wu. 2015
28 citations recorded by the source corpus.
In this paper we propose and study a technique to reduce the number of parameters in fully-connected layers of neural networks using Kronecker product, at a mild cost of the prediction quality. The technique proceeds by replacing fully-connected layers with so-called Kronecker fully-connected layers, where the weight matrices of the fully-connected layers are approximated by linear combinations of multiple Kronecker products of smaller matrices. Just as the Kronecker product is a generalization of the outer product from vectors to matrices, our method is a generalization of the low rank approximation method for fully-connected layers. We also use combinations of different shapes of Kronecker product to increase modelling capacity. Experiments on SVHN, scene text recognition and ImageNet dataset demonstrate that we can achieve 10x reduction of number of parameters with less than 1% drop in accuracy, showing the effectiveness and efficiency of our method.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
In Model Architectures
In Efficiency & Compression
Citations in the corpus
- Compressing RNNs for IoT devices by 15-38x using Kronecker Products (2019, methodology): “We deviate from (Zhou & Wu, 2015) by using KPs to compress RNNs and, instead of learning the decomposition for fixed RNN layers, we learn the KP factors directly.”