Training Neural Networks for and by Interpolation

By Leonard Berrada, Andrew Zisserman, M. Pawan Kumar. 2019

72 citations recorded by the source corpus.

The majority of modern deep learning models are able to interpolate the data: the empirical loss can be driven near zero on all samples simultaneously. In this work, we explicitly exploit this interpolation property for the design of a new optimization algorithm for deep learning. Specifically, we use it to compute an adaptive learning-rate given a stochastic gradient direction. This results in the Adaptive Learning-rates for Interpolation with Gradients (ALI-G) algorithm. ALI-G retains the advantages of SGD, which are low computational cost and provable convergence in the convex setting. But unlike SGD, the learning-rate of ALI-G can be computed inexpensively in closed-form and does not require a manual schedule. We provide a detailed analysis of ALI-G in the stochastic convex setting with explicit convergence rates. In order to obtain good empirical performance in deep learning, we extend the algorithm to use a maximal learning-rate, which gives a single hyper-parameter to tune. We show that employing such a maximal learning-rate has an intuitive proximal interpretation and preserves all convergence guarantees. We provide experiments on a variety of architectures and tasks: (i) learning a differentiable neural computer; (ii) training a wide residual network on the SVHN data set; (iii) training a Bi-LSTM on the SNLI data set; and (iv) training wide residual networks and densely connected networks on the CIFAR data sets. We empirically show that ALI-G outperforms adaptive gradient methods such as Adam, and provides comparable performance with SGD, although SGD benefits from manual learning rate schedules. We release PyTorch and Tensorflow implementations of ALI-G as standalone optimizers that can be used as a drop-in replacement in existing code (code available at this https URL ).

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Optimization & Training

Citations in the corpus

Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solution (2022, background): “First attempts on providing an efﬁcient variant of the stepsize that works well in the stochastic setting were made in [3, 24].”
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence (2020, methodology): “In the ALI-G algorithm proposed by Berrada et al. (2020), the step-size is set as: γk = min { fi(x k) ‖∇fi(xk)‖2+δ , η } , where δ > 0 is a positive constant.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, background): “[10] propose adaptive step-sizes for SGD on convex, finite-sum loss functions under an ✏-interpolation condition.”
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize (2021, unknown): “More recently, variations of the Polyak stepsize have been proposed for stochastic optimization (Loizou et al., 2021; Prazeres & Oberman, 2021; Berrada et al., 2020; Gower et al., 2021).”
Stochastic Adaptive Line Search for Differentially Private Optimization (2020, background): “Recent works [11], [17] have shown that combining SGD with line-search achieves fast convergence for both convex and non-convex problems, and is robust to the precise choices of hyperparameters, for over-parameterized models, with the price of additional objective evaluation (feed-forward steps for ”