Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer

By Yerlan Idelbayev, M. A. Carreira-Perpiñán. 2020

194 citations recorded by the source corpus.

Neural net compression can be achieved by approximating each layer's weight matrix by a low-rank matrix. The real difficulty in doing this is not in training the resulting neural net (made up of one low-rank matrix per layer), but in determining what the optimal rank of each layer is—effectively, an architecture search problem with one hyperparameter per layer. We show that, with a suitable formulation, this problem is amenable to a mixed discrete-continuous optimization jointly over the ranks and over the matrix elements, and give a corresponding algorithm. We show that this indeed can select ranks much better than existing approaches, making low-rank compression much more attractive than previously thought. For example, we can make a VGG network faster than a ResNet and with nearly the same classification error.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Mikhail Khodak
2. Lester W. Mackey
3. Nicoló Fusi
4. Neil A. Tenenholtz
5. Michael W. Mahoney

In Computer Vision and Pattern Recognition

1. Mikhail Khodak
2. Lester W. Mackey
3. Nicoló Fusi
4. Neil A. Tenenholtz

In Efficiency & Compression

1. Mikhail Khodak
2. Lester W. Mackey
3. Nicoló Fusi
4. Neil A. Tenenholtz

Citations in the corpus

• Initialization and Regularization of Factorized Neural Layers (2021, methodology): “…compression, albeit usually for factorizing pre-trained models (Nakkiran et al., 2015; Yaguchi et al., 2019; Yang et al., 2020) rather than low-rank initialization for end-to-end training; FD has been used in model compression using an uncompressed teacher (Idelbayev & Carreira-Perpi˜n´an, 2020).”