Pruning neural networks: is it time to nip it in the bud?

By Elliot J. Crowley, Jack Turner, Amos J. Storkey, Michael F. P. O'Boyle. 2018

31 citations recorded by the source corpus.

Pruning is a popular technique for compressing a neural network: a large pre-trained network is fine-tuned while connections are successively removed. However, the value of pruning has largely evaded scrutiny. In this extended abstract, we examine residual networks obtained through Fisher-pruning and make two interesting observations. First, when time-constrained, it is better to train a simple, smaller network from scratch than prune a large network. Second, it is the architectures obtained through the pruning process --- not the learnt weights ---that prove valuable. Such architectures are powerful when trained from scratch. Furthermore, these architectures are easy to approximate without any further pruning: we can prune once and obtain a family of new, scalable network architectures for different memory requirements.

Subfields: Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

1. Yi-Lin Sung
2. Varun Nair
3. Colin Raffel
4. Mohit Bansal
5. Jaemin Cho

Citations in the corpus

• Training Neural Networks with Fixed Sparse Masks (2021, background): “…learning, e.g. as a measure of parameter importance [21], as a preconditioner in gradient descent [4, 37, 34], as a way to measure the amount of “information” in each parameter of a neural network [1], or as a way to decide which parameters to prune when performing model compression [43, 10, 47].”