Overtraining, regularization and searching for a minimum, with application to neural networks

By Jonas Sjöberg, Lennart Ljung. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

A Closer Look at Memorization in Deep Networks (2017, methodology): “Sjoberg et al. (1995) showed that, for linear models, SGD ﬁrst learns models with small 2 parameter norm.”
Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm (2017, methodology): “Each time the model converges as measured on the validation set, the weights are reloaded to the best setting, thereby preventing overﬁtting in a similar manner to early stopping (Sj¨oberg and Ljung, 1995).”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “…the mode of an implicit posterior over φ resulting from an empirical loss interpreted as a negative log-likelihood, and regularization penalties and the early stopping procedure jointly acting as priors (for similar interpretations, see Sjöberg & Ljung, 1995; Bishop, 1995; Duvenaud et al., 2016).”
Deep learning for cellular image analysis (2019, background): “Because overfitting often gets worse the longer that training proceeds, stopping training early can also be effective 178 .”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, result): “However, we can still frame the effect of early stopping and initialization as serving as a prior in a similar way as prior work (Sj¨oberg & Ljung, 1995; Duvenaud et al., 2016; Grant et al., 2018).”
Assessing Shape Bias Property of Convolutional Neural Networks (2018, unknown): “zation error [1]. In [7], however, the authors argued that networks trained with SGD learn structures before memorizing. This follows the works suggesting that SGD generally results in simpler models [12,13]. Deep neural networks are known to be capable of approximating any measurable function given”