Auto-Sizing Neural Networks: With Applications to n-gram Language Models

By Kenton Murray, David Chiang. 2015

47 citations recorded by the source corpus.

Neural networks have been shown to improve performance across a range of natural-language tasks. However, designing and training them can be complicated. Frequently, researchers resort to repeated experimentation to pick optimal settings. In this paper, we address the issue of choosing the correct number of units in hidden layers. We introduce a method for automatically adjusting network size by pruning out hidden units through‘1,1 and ‘2,1 regularization. We apply this method to language modeling and demonstrate its ability to correctly choose the number of hidden units while maintaining perplexity. We also include these models in a machine translation decoder and show that these smaller neural models maintain the significant improvements of their unpruned versions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Neural Architecture Search & AutoML

Citations in the corpus

Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Structured pruning has been used in some NLP applications, such as machine translation (See et al., 2016), text classification (Joulin et al., 2016) and language modeling (Murray & Chiang, 2015).”
Are Sixteen Heads Really Better than One? (2019, methodology): “In NLP , structured pruning for auto-sizing feed-forward language models was ﬁrst investigated by Murray and Chiang (2015).”
MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “Except for the dropout, all other hyperparameters used at training were identical for all DNNs.”
Learning the Number of Neurons in Deep Networks (2016, background): “However, [Zhou et al., 2016] focuses on the last fully-connected layer to obtain a compact model, and [S. Tu, 2014] and [Murray and Chiang, 2015] only considered small networks.”
Compression of Neural Machine Translation Models via Pruning (2016, background): “Other approaches focus on pruning neurons rather than parameters, via sparsity-inducing regularizers (Murray and Chiang, 2015) or ‘wiring together’ pairs of neurons with similar input weights (Srinivas and Babu, 2015).”
Automatic node selection for Deep Neural Networks using Group Lasso regularization (2016, methodology): “In [14], the gLasso regularizer was defined for the incoming vector grouping(1) of neural network-based language models and its effect was evaluated for shallow networks.”
Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors (2018, unknown): “Closer to our work in spirit, (Ochiai et al., 2016; Scardapane et al., 2017; Alvarez & Salzmann, 2016) and (Murray & Chiang, 2015) who use group sparsity to prune groups of weights—e.g. weights incident to a node.”
Block Pruning For Faster Transformers (2021, background): “In contrast, structured pruning removes coherent groups of weights (Murray and Chiang, 2015; See et al., 2016; Joulin et al., 2016; Fan et al., 2020; Sajjad et al., 2020).”
Model Selection in Bayesian Neural Networks via Horseshoe Priors (2017, background): “Closest to our work in spirit, [24], [28] and [22] who use group sparsity to prune groups of weights—e.g. weights incident to a node.”
Runtime Neural Pruning (2017, background): “For example, [27, 44] introduced structured sparsity regularizers on the network weights, [32] put them to the hidden units.”