Early Stopping without a Validation Set

By Maren Mahsereci, Lukas Balles, Christoph Lassner, Philipp Hennig. 2017

113 citations recorded by the source corpus.

Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing estimate of the generalization performance. We propose a novel early stopping criterion based on fast-to-compute local statistics of the computed gradients and entirely removes the need for a held-out validation set. Our experiments show that this is a viable approach in the setting of least-squares and logistic regression, as well as neural networks.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

AutoML: A Survey of the State-of-the-Art (2019, methodology): “Furthermore, [127] present a novel approach of early stopping based on fast-to-compute local statistics of the computed gradients, which no longer relies on the validation set as previous methods, but can allow the optimizer to make full use of all training data.”
The Limitations of Federated Learning in Sybil Settings (2020, methodology): “For better training efficiency, the process may also use a dynamic early stopping heuristic, typically based on the norm of the gradients [37], or the validation error [47].”
Follow the Signs for Robust Stochastic Optimization (2017, background): “Several recent papers [11, 2, 12] have pointed out that an explicit, local estimate of the gradient variance r(θ) can be obtained from a single mini-batch evaluation via r̂(θ) = 1 |B| − 1 (”