Learning the Number of Neurons in Deep Networks

By José Manuel Álvarez, Mathieu Salzmann. 2016

427 citations recorded by the source corpus.

Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80\% while retaining or even improving the network accuracy.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Recurrent Neural Networks for Semantic Instance Segmentation (2017, background): “There is evidence that the optimal number of units is very dependent on the dataset [42], but models with more parameters are also slower to train and require more memory.”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “The network is pruned with simple methods like L1 norm and more compression tricks like vector quantization [17] and group sparsity regularization [1] can be applied.”
To prune, or not to prune: exploring the efficacy of pruning for model compression (2017, methodology): “ize include low-rank matrix factorization (Denil et al., 2013; Denton et al., 2014; Jaderberg et al., 2014; Lebedev et al., 2014) and group sparsity regularization to arrive at an optimal layer size (Alvarez and Salzmann, 2016). 3 Methods We extend the TensorFlow (Abadi et al., 2015) framework to pr”
Network Pruning via Transformable Architecture Search (2019, methodology): “, pruning 30% filters in each layer [27, 15], predicting sparsity ratio [12], or leveraging regularization [2].”
Learning to Prune Filters in Convolutional Neural Networks (2018, background): “[1, 35, 28] also proposed to remove the neurons in networks to increase the sparsity.”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “Except for the dropout, all other hyperparameters used at training were identical for all DNNs.”
Lifelong Learning with Dynamically Expandable Networks (2017, positive): “This group sparsity regularization was used in Wen et al. (2016) and Alvarez & Salzmann (2016) to ﬁnd the right number of neurons for a full network, while we apply it to the partial network.”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, methodology): “Recently, training-based channel pruning works [1, 49] have focused on imposing sparse constrain on weights during training, which could adaptively determine hyper-parameters.”
CondenseNet: An Efficient DenseNet Using Learned Group Convolutions (2017, methodology): “In contrast, coarse-grained pruning methods such as filter-level pruning [1, 14, 29, 32] achieve a lower degree of sparsity, but the resulting networks are much more regular, which facilitates efficient implementations.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, methodology): “[45, 1] added group sparsity regularizations on the weights of neurons and sparsified them in the training stage.”
Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers (2018, background): “Some recent efforts proposed to impose structured sparsity on those vector components motivated from the implementation perspective on specialized hardware (Wen et al., 2016; Zhou et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016).”
Discrimination-aware Channel Pruning for Deep Neural Networks (2018, methodology): “Or equivalently, given a predefined pruning rate η ∈ (0, 1) [1, 25], it follows that κl = dηce.”
Domain-Adaptive Deep Network Compression (2017, methodology): “Approaches include methods based on weight quantization [13, 27], weight removal from fully-connected [38] or convolutional layers [2], compact representations of convolutional layers through tensor decompositions [7, 18, 1], as well as training of thinner networks from predictions of a larger…”
Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks (2019, neutral): “4) Filter pruning [1, 30, 15, 25, 28, 33, 56, 52].”
ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks (2018, background): “While compression is typically performed as a post-processing step, it has been shown that incorporating it during training could be beneficial [3, 4, 56, 57].”
Auto-Balanced Filter Pruning for Efficient Convolutional Neural Networks (2018, background): “2016) (Alvarez and Salzmann 2016) (Lebedev and Lempitsky 2016).”