Group sparse regularization for deep neural networks

By Simone Scardapane, Danilo Comminiello, Amir Hussain, Aurelio Uncini. 2016

500 citations recorded by the source corpus.

In this paper, we address the challenging task of simultaneously optimizing (i) the weights of a neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are traditionally dealt with separately, we propose an efficient regularized formulation enabling their simultaneous parallel execution, using standard optimization routines. Specifically, we extend the group Lasso penalty, originally proposed in the linear regression literature, to impose group-level sparsity on the networks connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We carry out an extensive experimental evaluation, in comparison with classical weight decay and Lasso penalties, both on a toy dataset for handwritten digit recognition, and multiple realistic mid-scale classification benchmarks. Comparative results demonstrate the potential of our proposed sparse group Lasso penalty in producing extremely compact networks, with a significantly lower number of input features, with a classification accuracy which is equal or only slightly inferior to standard regularization terms.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual and Multi-Task Architecture Search (2019, neutral): “Scardapane et al. (2017) proposed group sparsity in the”
Continual Learning with Node-Importance based Adaptive Group Sparse Regularization (2020, methodology): “The group Lasso [40] regularization, which was favorably used in model compression [4, 37, 38, 29], has been also adopted for continual learning in [39].”
Variational Dropout Sparsifies Deep Neural Networks (2017, background): “Recent works on Bayesian DNNs (Kingma & Welling, 2013; Rezende et al., 2014; Scardapane et al., 2016) provide different ways to train deep models with a huge number of parameters in a Bayesian way.”
Learning Efficient Convolutional Networks through Network Slimming (2017, background): “Sparsity can be imposed on different level of structures [2, 37, 35, 29, 25], which yields considerable model-size compression and inference speedup.”
Bayesian Compression for Deep Learning (2017, methodology): “The mode of the posterior distribution under a Laplace prior is known as the Lasso [69] estimator and has been previously used for sparsifying neural networks at [73, 61].”
Learning Explainable Models Using Attribution Priors (2019, background): “This can be extended in the sparse group lasso (SGL) to penalize all weights connected to a given feature [8, 30].”
Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes (2016, background): “Scardapane et al. (2016) investigated mixed norms in order to promote group sparsity.”
SSN: Learning Sparse Switchable Normalization via SparsestMax (2019, methodology): “In [23], group Lasso penalty is adopted to impose group-level sparsity on network’s connections.”
RNN Architecture Learning with Sparse Regularization (2019, background): “…the structure of the model and only look at the value of individual weights (Liu et al., 2019; LeCun et al., 1990; Lee et al., 2019; Frankle and Carbin, 2019) or only use high-level structures like the number of layers of the network (Wen et al., 2016; Scardapane et al., 2017; Gordon et al., 2018).”
Differentiable Branching In Deep Networks for Fast Inference (2020, methodology): “Known ways of achieving low inference times include, among others, compression and pruning of the neural network weights [4], regularization [5], using low-precision arithmetic [6, 7], or exploiting ad-hoc components for fast inference [8].”
Automatic node selection for Deep Neural Networks using Group Lasso regularization (2016, methodology): “In [10], the gLasso regularizer was defined for the outgoing vector grouping of DNN and its effect was evaluated in various small- and middle-sized tasks that classify fixed-dimensional patterns.”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, background): “Closer to our work in spirit, (Ochiai et al., 2016; Scardapane et al., 2017; Alvarez & Salzmann, 2016) and (Murray & Chiang, 2015) who use group sparsity to prune groups of weights—e.”
Adaptive Group Sparse Regularization for Continual Learning (2020, methodology): “Since identifying important nodes while training a neural network model is exactly what is done in the group Lassobased model compression methods [33, 3, 34, 26], we adopt the same methodology for compactly identifying important nodes when continuously learning new tasks.”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “Closest to our work in spirit, [24], [28] and [22] who use group sparsity to prune groups of weights—e.g. weights incident to a node.”
Improving performance of deep learning models with axiomatic attribution priors and expected gradients (2020, methodology): “While feature attributions have previously been used in training (more details in Methods 1) [34, 19], our approach offers two novel components: First, we demonstrate that calculating Φ with attribution methods that satisfy previouslyestablished interpretability axioms improves performance (see Sect”
A Framework for Parallel and Distributed Training of Neural Networks (2016, background): “To this aim, we build a surrogate function g̃i that additively decomposes over the different cores, i.e.: g̃i(wi; wi[n]) = C∑ c=1 g̃i,c(wi,c; wi,−c[n]) , (34) where each g̃i,c(·; wi,−c[n]) is any surrogate function satisfying conditions F1F3 on the variable wi,c.”