Early Stopping is Nonparametric Variational Inference

By Dougal Maclaurin, David Kristjanson Duvenaud, Ryan P. Adams. 2015

100 citations recorded by the source corpus.

We show that unconverged stochastic gradient descent can be interpreted as a procedure that samples from a nonparametric variational approximate posterior distribution. This distribution is implicitly defined as the transformation of an initial distribution by a sequence of optimization updates. By tracking the change in entropy over this sequence of transformations during optimization, we form a scalable, unbiased estimate of the variational lower bound on the log marginal likelihood. We can use this bound to optimize hyperparameters instead of using cross-validation. This Bayesian interpretation of SGD suggests improved, overfitting-resistant optimization procedures, and gives a theoretical foundation for popular tricks such as early stopping and ensembling. We investigate the properties of this marginal likelihood estimator on neural network models.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Early Stopping without a Validation Set (2017, background): “2), leaving less effective training data to define the training loss LD.”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “We can instead understand the point estimate given by truncated gradient descent as the value of the mode of an implicit posterior over φ resulting from an empirical loss interpreted as a negative log-likelihood, and regularization penalties and the early stopping procedure jointly acting as priors ”
Stochastic Gradient Descent as Approximate Bayesian Inference (2017, background): “Maclaurin et al. (2016) also interpret SGD as a non-parametric variational inference scheme, but with different goals and in a different formalism.”
Neural Autoregressive Flows (2018, background): “Finally, there are also several works which develop new techniques for constructing NFs that are orthogonal to ours (Tomczak & Welling, 2017; 2016; Gemici et al., 2016; Duvenaud et al., 2016; Berg et al., 2018).”
Low-N protein engineering with data-efficient deep learning (2020, methodology): “We initialized the weights of the 1900 dimensional UniRep mLSTM with the globally pre-trained weights and trained for 13,500 iterations with early stopping 71,72 , until the outer validation set loss began to increase.”
Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models (2020, neutral): “This has been previously explored by Duvenaud et al. (2016).”
End-to-End Learning for Structured Prediction Energy Networks (2017, methodology): “The method of Wang et al. (2016) cannot handle this prior.”
Advances in Variational Inference (2017, methodology): “Here we consider SGD with constant learning rates (constant SGD) [113], [114], and early stopping [43].”
Learning a Prior over Intent via Meta-Inverse Reinforcement Learning (2018, result): “However, we can still frame the effect of early stopping and initialization as serving as a prior in a similar way as prior work (Sj¨oberg & Ljung, 1995; Duvenaud et al., 2016; Grant et al., 2018).”
Wild Variational Approximations (2016, background): “(Stochastic gradient descent (SGD) with constant/adaptive learning rates) It has been shown in [23, 24] that the trajectory of SGD with constant/adaptive learning rates near a local optimum can be viewed as a variational approximation to the exact posterior.”
Pathfinder: Parallel quasi-Newton variational inference (2021, unknown): “Early stopping variational inference generates approximate posterior draws by taking random draws from an initialization distribution, then following an optimization path for a fixed number of steps and taking the result as an approximate posterior draw (Duvenaud et al., 2016).”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, background): “The tendency for SGD w/ early-stopping to self-regularize is well-documented, and there are many compelling theories about why this happens (e.g. Duvenaud et al., 2016; Hardt et al., 2015).”
Two Methods for Wild Variational Inference (2016, methodology): “…work: one based on “amortizing SVGD” (Wang & Liu, 2016) which trains the inference network f(η, ξ) so that its output mimic the SVGD dynamics in order to decrease the KL divergence; another based on minimizing the KSD objective (9) which does not require to evaluate q(z) thanks to its special form.”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “It will be presented at AISTATS in May 2016 [25].”