On Early Stopping in Gradient Descent Learning

By Yuan Yao, Lorenzo Rosasco, Andrea Caponnetto. 2007

1,149 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, methodology): “This early stopping strategy is widely used in practice, and has been studied theoretically (Chauvin, 1990; P.F. Baldi and Hornik, 1995; Dodier, 1995; Amari, Murata, Müller, et al., 1995; Amari, Murata, and Müller, 1996; Biehl, Caticha, and Riegler, 2009; Yao, Rosasco, and Caponnetto, 2007).”
A Closer Look at Memorization in Deep Networks (2017, background): “More generally, the efﬁcacy of early stopping shows that SGD ﬁrst learns simpler models (Yao et al., 2007).”
Federated Optimization in Heterogeneous Networks (2018, background): “Indeed, it is wellknown that solving the problem beyond some threshold may even hurt generalization performance due to overfitting (Yao et al., 2007).”
Federated Learning: Challenges, Methods, and Future Directions (2019, background): “Indeed, it is well known that optimization methods used for machine learning can tolerate a lack of precision; this error can, in fact, help with generalization [49].”
Recent advances in convolutional neural networks (2015, methodology): “le GPU. 15 Note that SGD methods may not result in convergence. The training process can be terminated when the performance stops improving. A popular remedy to over-training is to use early stopping [117] in which optimization is halted based on the performance on a validation set during training. ”
The Marginal Value of Adaptive Gradient Methods in Machine Learning (2017, background): “Most of the seminal work in this area has focused on understanding how early stopping can act as implicit regularization [22].”
On Lazy Training in Differentiable Programming (2018, background): “To ensure that the bad performances corresponding to large τ are not due to a lack of regularization, we display also the best test error throughout training (for kernel methods, early stopping is a form of regularization [34]).”
Divide and conquer kernel ridge regression: a distributed algorithm with minimax optimal rates (2013, background): “A second line of research has considered early-stopping of iterative optimization algorithms for KRR, including gradient descent (Yao et al., 2007; Raskutti et al., 2011) and conjugate gradient methods (Blanchard and Kr¨amer, 2010), where early-stopping provides regularization against over-ﬁtting…”
Just Interpolate: Kernel "Ridgeless" Regression Can Generalize (2018, methodology): “[30] Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto.”
To understand deep learning we need to understand kernel learning (2018, methodology): “ularization introduced by methods such as early stopping in gradient descent (Yao et al., 2007; Raskutti et al., 2014; Neyshabur et al., 2014; Camoriano et al., 2016).”
Self-Balancing Federated Learning With Global Imbalanced Data in Mobile Systems (2021, methodology): “To remedy the loss of accuracy caused by overfitting, we can use the regularization strategy called early stopping [46], in which optimization is halted based on the performance on a validation set, during training.”
Revisiting Challenges in Data-to-Text Generation with Fact Grounding (2020, methodology): “(2017), we split all samples into train (70%), validation (15%), and test (15%) sets, and perform early stopping (Yao et al., 2007) using BLEU (Papineni et al.”
Robust early-learning: Hindering the memorization of noisy labels (2021, methodology): “It was used in supervised learning early (Prechelt, 1998; Caruana et al., 2001; Zhang et al., 2005; Yao et al., 2007).”
Mercer’s Theorem on General Domains: On the Interaction between Measures, Kernels, and RKHSs (2012, methodology): “We begin with a result on the Fourier coefficients of fP,λ, which generalizes an analogous formula from [46] for continuous kernels on compact metric spaces.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “This ﬁnding lets us rethink the role of “early stopping” (Yao et al., 2007; Arpit et al., 2017) on real-world noisy data.”
Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks (2020, methodology): “The (Yao, Rosasco, and Caponnetto 2007) is applied when the validation loss stops decreasing by 10 epochs.”