DropNeuron: Simplifying the Structure of Deep Neural Networks

By W. Pan, Hao Dong, Yike Guo. 2016

40 citations recorded by the source corpus.

Deep learning using multi-layer neural networks (NNs) architecture manifests superb power in modern machine learning systems. The trained Deep Neural Networks (DNNs) are typically large. The question we would like to address is whether it is possible to simplify the NN during training process to achieve a reasonable performance within an acceptable computational time. We presented a novel approach of optimising a deep neural network through regularisation of net- work architecture. We proposed regularisers which support a simple mechanism of dropping neurons during a network training process. The method supports the construction of a simpler deep neural networks with compatible performance with its simplified version. As a proof of concept, we evaluate the proposed method with examples including sparse linear regression, deep autoencoder and convolutional neural network. The valuations demonstrate excellent performance. The code for this work can be found in this http URL

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Efficient Structured Pruning and Architecture Searching for Group Convolution (2018, methodology): “Some other methods use group LASSO [48] to encode specific structures during regularisation, such as channel or filter level pruning [30, 41, 18].”
Efficient Document Re-Ranking for Transformers by Precomputing Term Representations (2020, background): “. Pruning techniques work both at learning time and as a post-learning step. In the first category, Pan et al. propose regularization techniques focused at removing redundant neurons at training time [33]. Alternatively, in the second category, Han et al. propose to remove the smallest weights in te”
Compressing Neural Networks using the Variational Information Bottleneck (2018, methodology): “…(Louizos et al., 2017a), Sparse ‘ 0 Regularization (L0) and L0 with separate λ for each layer (L0-sep) (Louizos et al., 2017b), Drop Neuron (DN) (Pan et al., 2016), Runtime Neural Pruning (RNP) (Lin et al., 2017), Pruning Filter (PF) (Li et al., 2016), Network Slimming (NS) (Liu et al.,…”
Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, methodology): “Some methods add to the training loss a regularization function such as L1 [21], Group LASSO [33], or an approximation of the L0 norm [22, 28].”