Generalization and Parameter Estimation in Feedforward Netws: Some Experiments

By Nelson Morgan, Hervé Bourlard. 1989

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Early Stopping without a Validation Set (2017, methodology): “Finally, a widely-used provision against overfitting is to add regularization terms to the objective function that penalize the parameter vector w, typically measured by the l1 or l2 norm [9].”
A Survey of Deep Learning Applications to Autonomous Vehicle Control (2019, background): “Also, given that additional neurons will lead to increased computational complexity, finding an optimal number of neurons would be of great benefit to deep learning methods [158], [159].”
Early Stopping-But When? (1996, background): “Techniques for reducing the size of each parameter dimension are regularization, such as weight decay [13] and others [25], or early stopping [17].”
Automatic early stopping using cross validation: quantifying the criteria (1998, methodology): “The corresponding NN techniques for reducing the size of each parameter dimension are regularization such as weight decay (e.g. Krogh & Hertz, 1992) and others (e.g. Weigend, Rumelhart & Huberman, 1991) or early stopping (Morgan & Bourlard, 1990).”
Early stopping for non-parametric regression: An optimal data-dependent stopping rule (2011, unknown): “ber iteration. (For instance, see the paper by Strand [35], with follow-up work by Anderssen and Prenter [1] as well as Wahba [38].) Early stopping has also been widely used in neural networks (e.g., [28]), in which stochastic gradient descent is used to estimate the network parameters. Past work pr”
Brain age prediction using deep learning uncovers associated sequence variants (2019, methodology): “Early stopping 67 with model checkpointing was used, i.e., if the validation error did not improve in 100 epochs the training was stopped and the weights with the lowest validation error selected.”
Multivariate quantile mapping bias correction: an N-dimensional probability density function transform for climate model simulations of multiple variables (2017, background): “To help avoid overfitting for high dimensional problems (or more generally whenever N is large relative to I), early stopping (Morgan and Bourlard 1990; Prechelt 1998)—an implicit form of regularization commonly used in iteratively-trained machine learning methods—may be effective for MBCn.”
Overfitting in adversarially robust deep learning (2020, methodology): “…regularization (also known as weight decay) is commonly used for training deep networks (Krogh & Hertz, 1992), early stopping is less commonly used despite being studied as an implicit regularizer for controlling model complexity for neural networks at least 30 years ago (Morgan & Bourlard, 1990).”
Neural networks for classification: a survey (2000, methodology): “Such methods include cross validation [118], [184], training with penalty terms [182], and weight decay and node pruning [137], [148].”
Deep learning generalizes because the parameter-function map is biased towards simple functions (2018, background): “Regarding 1), it was originally thought that regularization methods such as Tikhonov regularization (Tikhonov (1943)), dropout (Srivastava et al. (2014)), or early stopping (Morgan & Bourlard (1990)) were key in providing this inductive bias.”
BERT Loses Patience: Fast and Robust Inference with Early Exit (2020, methodology): “mechanism [7, 8].”
Geometry of Early Stopping in Linear Networks (1995, background): “The earliest references include Morgan and Bourlard [4] and Weigend et al.”
Neural Networks and the Bias/Variance Dilemma (1992, result): “This tradeoff is, in fact, observed in experiments reported by several authors (see, for example, Chauvin 1990; Morgan and Bourlard 1990), as well as in our experiments with artificial data (Section 3.”