Learning Sparse Neural Networks via Sensitivity-Driven Regularization

By Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, Gianluca Francini. 2018

74 citations recorded by the source corpus.

The ever-increasing number of parameters in deep neural networks poses challenges for memory-limited applications. Regularize-and-prune methods aim at meeting these challenges by sparsifying the network weights. In this context we quantify the output sensitivity to the parameters (i.e. their relevance to the network output) and introduce a regularization term that gradually lowers the absolute value of parameters with low sensitivity. Thus, a very large fraction of the parameters approach zero and are eventually set to zero by simple thresholding. Our method surpasses most of the recent techniques both in terms of sparsity and error rates. In some cases, the method reaches twice the sparsity obtained by other techniques at equal error rates.

Subfields: Artificial Intelligence, Computational Mechanics, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Greg Mori
2. Yi Yang
3. Ping Luo
4. Xiaogang Wang
5. Mengyao Zhai
6. Lei Chen
7. Frederick Tung
8. Xuanyi Dong

In Computational Mechanics

1. Greg Mori
2. Xuanyi Dong
3. Wenqi Shao
4. Erich Elsen
5. Frederick Tung
6. Ruimao Zhang
7. Xiaogang Wang
8. Pablo Samuel Castro

In Statistical and Nonlinear Physics

1. Greg Mori
2. Wenqi Shao
3. Erich Elsen
4. Trevor Gale
5. Xiaogang Wang
6. Ping Luo
7. Yi Yang
8. Megha Nawhal

Citations in the corpus

• Network Pruning via Transformable Architecture Search (2019, background): “After this development, many researchers explored different regularization techniques to improve the sparsity while preserve the accuracy, such as L0 regularization [32] and output sensitivity [38].”
• Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation (2021, background): “Parameter efficiency can also be encouraged during training via architecture design [23,29], sparsity inducing priors [33], resource re-allocation [26], or knowledge distillation from a teacher network [34,41].”
• Rigging the Lottery: Making All Tickets Winners (2019, methodology): “More recent work includes L 0 Regularization (Chris-tos Louizos, 2018), Variational Dropout (Molchanov et al., 2017), Dynamic Network Surgery (Guo et al., 2016), Discovering Neural Wirings (Wortsman et al., 2019), Sensitivity Driven Regularization (Tartaglione et al., 2018).”
• SSN: Learning Sparse Switchable Normalization via SparsestMax (2019, methodology): “To reduce computation, sparse learning arewidely performed to compress the scale ofDNN, including work on sparsity regularization (Liu et al. 2015; Tartaglione et al. 2018) and low-rank regularization (Tai et al.”