Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping

By R. Caruana, Steve Lawrence, C. Lee Giles. 2000

1,197 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, neutral): “Our findings agree both with early numerical studies of neural networks (Caruana, Lawrence, and Giles, 2001) and the good generalization performance of massive neural networks in recent years (e.g. Simonyan and Zisserman, 2015).”
Learning to Answer Questions from Image Using Convolutional Neural Network (2015, methodology): “In order to prevent overﬁtting, eartly stopping (Caruana et al., 2000) and dropout (with probability 0.1) are employed.”
Understanding Membership Inferences on Well-Generalized Learning Models (2018, background): “In machine learning, one way of preventing overfitting is to stop training the model as soon as the testing accuracy stops increasing [8].”
Reversible Recursive Instance-Level Object Segmentation (2015, methodology): “To obtain a better refined bounding box for each proposal, the proposal refinement sub-network adaptively determines the number of iterations for refining each proposal in both training and testing, which is in spirit similar to the early stopping rules for iteratively training large networks [6].”
On the Efficacy of Knowledge Distillation (2019, background): “There is some evidence that a large network trained with only a few epochs behaves as a small network, while still encompassing a greater search space than small network [3, 18].”
A jamming transition from under- to over-parametrization affects loss landscape and generalization (2018, background): “The former displays a cusp at the transition point, indicating over-fitting [7].”
Machine Learning for 5G/B5G Mobile and Wireless Communications: Potential, Limitations, and Future Directions (2019, neutral): “This ability is called generalization [37].”
Building DNN acoustic models for large vocabulary speech recognition (2014, background): “[67] found that early stopping training of networks with large capacity produces generalization performance on par with or better than the generalization of a smaller network.”