Learning Sparse Neural Networks through L0 Regularization

By Christos Louizos, Max Welling, Diederik P. Kingma. 2017

1,332 citations recorded by the source corpus.

We propose a practical method for $L_0$ norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularization is interesting since (1) it can greatly speed up training and inference, and (2) it can improve generalization. AIC and BIC, well-known model selection criteria, are special cases of $L_0$ regularization. However, since the $L_0$ norm of weights is non-differentiable, we cannot incorporate it directly as a regularization term in the objective function. We propose a solution through the inclusion of a collection of non-negative stochastic gates, which collectively determine which weights to set to zero. We show that, somewhat surprisingly, for certain distributions over the gates, the expected $L_0$ norm of the resulting gated weights is differentiable with respect to the distribution parameters. We further propose the \emph{hard concrete} distribution for the gates, which is obtained by "stretching" a binary concrete distribution and then transforming its samples with a hard-sigmoid. The parameters of the distribution over the gates can then be jointly optimized with the original network parameters. As a result our method allows for straightforward and efficient learning of model structures with stochastic gradient descent and allows for conditional computation in a principled way. We perform various experiments to demonstrate the effectiveness of the resulting approach and regularizer.

Subfields: Artificial Intelligence, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Efficiency & Compression

Citations in the corpus

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, background): “, 2020), model structures/sparsities (Louizos et al., 2018; Gong et al., 2019; Tan & Le, 2019), backpropagation storage requirements (Gruslys et al.”
The State of Sparsity in Deep Neural Networks (2019, methodology): “At test-time, Louizos et al. (2017b) use the following estimate for the model parameters.”
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints (2022, methodology): “On the other hand, in-training methods which learn a good sparsity pattern by augmenting the training loss with sparsity-inducing penalties [31, 27] do not perform fine-tuning, but face challenges regarding the tuning and interpretability of the penalty hyperparameter.”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “Our method is based on Louizos et al. (2018).”
Training Neural Networks with Fixed Sparse Masks (2021, methodology): “Mask sparsity in the binary mask is enforced through a smooth approximation of the L 0 norm introduced by Louizos et al. [31].”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, background): “, 2013) or learn per-weight dropout probabilities (Molchanov et al., 2017; Louizos et al., 2018) could make winning tickets even easier to find.”
SNAS: Stochastic Neural Architecture Search (2018, methodology): “Apart from ENAS, we are also inspired by Louizos et al. (2017) to use continuous distribution for structural decision at each edge and optimize it along with an l 0 complexity regularizer.”
Dynamic Model Pruning with Feedback (2020, background): “Gale et al. (2019) extensively study recent unstructured pruning methods on large-scale learning tasks, and ﬁnd that complex techniques (Molchanov et al., 2017; Louizos et al., 2018) perform inconsistently.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “To introduce a selection bottleneck, we impose a regularization on the expected L0 norm of Z = (z1 · · · zT−1) to promote sparsity at a targeted value μ0 (Louizos et al., 2017; Basting & et al, 2019).”
Network Pruning via Transformable Architecture Search (2019, background): “After this development, many researchers explored different regularization techniques to improve the sparsity while preserve the accuracy, such as L0 regularization [32] and output sensitivity [38].”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, neutral): “…3-18-284-283 0.90 0.69 BC-GNJ ((Louizos et al., 2017)) 8-13-88-13 1.00 1.09 BC-GHS ((Louizos et al., 2017)) 5-10-76-16 1.00 0.57 Practical `0((Louizos et al., 2018)) 20-25-45-462 0.90 4.49 Practical `0 ((Louizos et al., 2018)) 9-18-65-25 1.00 1.55 Proposed method 5-10-65-25 1.00 0.57 Table…”
Rethinking the Value of Network Pruning (2018, methodology): “Unstructured pruning (Han et al., 2015; Molchanov et al., 2017; Louizos et al., 2018) also falls in the category of automatic methods, where the positions of pruned weights are determined by the training process and the pruning algorithm, and it is usually not possible to predefine the positions of ”
Adaptively Sparse Transformers (2019, background): “This has motivated a line of research on learning networks with sparse mappings (Martins and Astudillo, 2016; Niculae and Blondel, 2017; Louizos et al., 2018; Shao et al., 2019).”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, methodology): “This is less restrictive than traditional group sparsity techniques that have been used with L0-norm relaxations, which force all parameters in a group to share the same mask (Louizos et al., 2018; Wang et al., 2019b).”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Weight pruning is implemented during training (Louizos et al., 2017) or as a ﬁne-tuning post-processing step (Han et al., 2015; 2016).”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, methodology): “Following Louizos et al. (2017), we relax non-differentiable penalties by computing them on expectation under our latent model p ( z | x, φ ) .”
Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning (2020, background): “Sparse matrix multiplication is difﬁcult to optimize on current GPU architectures (Han et al., 2016), although progress is being made. a form of permanent dropout (Molchanov et al., 2017) or a heuristic-based L0 regularizer (Louizos et al., 2018).”
Sparse Networks from Scratch: Faster Training without Losing Performance (2019, methodology): “Other compression algorithms include L 0 regularization (Louizos et al., 2018), and Bayesian methods (Louizos et al., 2017; Molchanov et al., 2017).”