Simplifying Neural Networks by Soft Weight-Sharing

By Steven J. Nowlan, Geoffrey E. Hinton. 1992

One way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity. Simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights. We propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians. A set of weights is simple if the weights have high probability density under the mixture model. This can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values. Since we do not know the appropriate means or variances of the clusters in advance, we allow the parameters of the mixture model to adapt at the same time as the network learns. Simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Weight Agnostic Neural Networks (2019, background): “Ideas related to MDL for making neural networks “simple” was proposed in the 1990s, such as simplifying networks by soft-weight sharing [80], reducing the amount of information in weights by making them noisy [43], and simplifying the search space of its weights [95].”
Deep learning in neural networks: An overview (2014, background): “Some assume that a distribution of networks with many similar weights generated by Gaussian mixtures is “better” a priori (Nowlan and Hinton, 1992).”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, methodology): “…methods are also applied to network pruning (Ullrich et al. (2017); Molchanov et al. (2017)) where the former extends the soft weight sharing in Nowlan & Hinton (1992) to obtain a sparse and compressed network, and the latter uses variational inference to learn the dropout rate which can then…”
Auto-Sizing Neural Networks: With Applications to n-gram Language Models (2015, methodology): “Schmidhuber and Heil (1996) proposed a character n-gram model using neural networks which they used for text compression.”
Uncertainty in Deep Learning (2016, background): “Many works at the time tried to approximate inference in similarly motivated models to varying degrees of success [Ji et al., 1990; Nowlan and Hinton, 1992; Weigend et al., 1991], none of which survived to modern days.”
Augmenting Image Classifiers Using Data Augmentation Generative Adversarial Networks (2018, unknown): “Techniques have been developed over the years to help combat overfitting such as L1/L2 reqularization [28], dropout [20], batch normalization [23], batch renormalisation [22] or layer normalization [2].”
Exploring Energy and Accuracy Tradeoff in Structure Simplification of Trained Deep Neural Networks (2018, unknown): “Nowlan and Hinton [25] proposed a different approach called weight sharing where weights of similar values in the same weight matrix are grouped together so that those inputs that are to be multiplied to these grouped weights may be added first and then perform a single multiplication.”
Dropout: a simple way to prevent neural networks from overfitting (2014, background): “These include stopping the training as soon as performance on a validation set starts to get worse, introducing weight penalties of various kinds such as L1 and L2 regularization and soft weight sharing (Nowlan and Hinton, 1992).”
What Do Compressed Deep Neural Networks Forget (2019, background): “Work since the 1990s has shown that deep neural networks can be pruned of “excess capacity” in a similar fashion to synaptic pruning (Cun et al., 1990; Hassibi et al., 1993a; Nowlan & Hinton, 1992; Weigend et al., 1991).”
Extracting refined rules from knowledge-based neural networks (2004, methodology): “For instance, Nowlan and Hinton's (1991) training method, 'which groups links into loose clusters, may result in networks amenable to MoFN.”
Probabilistic Matrix Factorization (2007, methodology): “Introducing priors for the hyperparameters and max imizing the log-posterior of the model over both parameters a nd hyperparameters as suggested in [6] allows model complexity to be controlled automatically bas ed on the training data.”