Generalization by Weight-Elimination with Application to Forecasting

By Andreas S. Weigend, David E. Rumelhart, Bernardo A. Huberman. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep learning in neural networks: An overview (2014, background): “For example, weight decay (Hanson and Pratt, 1989; Weigend et al., 1991; Krogh and Hertz, 1992) prefers small weights and can be derived from Gaussian or Laplace weight priors (Hinton and van Camp, 1993); see also (Murray and Edwards, 1993).”
BayesNAS: A Bayesian Approach for Neural Architecture Search (2019, background): “The conventional approach to achieve sparsity is by enforcing penalty terms (Chauvin, 1989; Weigend et al., 1991; Ishikawa, 1996).”
Learning the Number of Neurons in Deep Networks (2016, methodology): “Such a direct approach has been employed by [Weigend et al., 1991] and [Collins and Kohli, 2014] who have developed regularizers that favor eliminating some of the parameters of the network, thus leading to lower memory requirement.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, background): “That is, large absolute weight values are considered unlikely and cost more bits than small ones [91, 294, 135, 97].”
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science (2017, neutral): “However, we do not exclude the possibility that there may be better, more sophisticated approaches to removing connections, e.g. using gradient methods 25 , or centrality metrics from network science 48 .”
SNIP: Single-shot Network Pruning based on Connection Sensitivity (2018, methodology): “The methods from the former category (Chauvin (1989); Weigend et al. (1991); Ishikawa (1996)) augment the loss function with some sparsity enforcing penalty terms ( e.g ., L 0 or L 1 norm), so that back-propagation effectively penalizes the magnitude of the weights during training.”
Evaluation of convolutional neural networks for visual recognition (1998, background): “Weight sharing can also be considered as an alternative to weight elimination [ 27 ] in order to reduce the number”
A Simple Weight Decay Can Improve Generalization (1991, background): “Often the number of free parameters, i. e. the number of weights and thresholds, is used as a measure of the network complexity, and algorithms have been developed, which minimizes the number of weights while still keeping the error on the training examples small [4] [5] [6].”
Forecasting exchange rate with deep belief networks (2011, result): “Weigend et al. [3], [ 4 ] compare the performance of neural network with that of random walk in predicting the Deutsche”
Flat Minima (1997, methodology): “…learning rate for the quadratic error. is the learning rate for the additional error term (the following values are used to make lambda updates according to Weigend et al., 1991, see also section 5.6). is a parameter needed for updating . is a parameter needed for updating , typical value is 0.5.”
Sales forecasting using extreme learning machine with applications in fashion retailing (2008, background): “Many studies conclude that ANN is better than various conventional methods [1,3,28,29,39].”
Early Stopping-But When? (1996, background): “Techniques for reducing the size of each parameter dimension are regularization, such as weight decay [13] and others [25], or early stopping [17].”
Selective Brain Damage: Measuring the Disparate Impact of Model Pruning (2019, methodology): “The paper was among the ﬁrst [27, 57, 68] to propose that deep neural networks could be pruned of “excess capacity” in a similar fashion to synaptic pruning.”
Support vector machines for SAR automatic target recognition (2001, methodology): “Here a cost functional with a weight decay [37] term is utilized in the experiments, J(w) = X i (y i ¡ g(w, x i )) 2 + ¸ ¢ kwk 2 : (9) The parameter¸has to be experimentally determined.”
A hybrid approach of neural network and memory-based learning to data mining (2000, methodology): “Detailed discussions on pruning and feature selection methods using neural networks are found in [ 13 ]‐[15].”