On the distance between two neural networks and the stability of learning

By Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu. 2020

75 citations recorded by the source corpus.

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems not to require learning rate grid search, it may unlock a simpler workflow for training deeper and more complex neural networks. Please find the Python code used in this paper here: this https URL.

Subfields: Scaling & Learning Theory, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

In Statistical Learning Theory

Citations in the corpus

High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “AGC is closely related to a recent line of work studying “normalized optimizers” (You et al., 2017; Bernstein et al., 2020; You et al., 2019), which ignore the scale of the gradient by choosing an adaptive learning rate inversely proportional to the gradient norm.”
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients (2020, methodology): “We performed extensive comparisons with other optimizers, including SGD [3], AdaBound [12], 466 Yogi [14], Adam [8], MSVAG [15], RAdam [16], Fromage [17] and AdamW [18].”
A modified Adam algorithm for deep neural network optimization (2023, methodology): “The MSVAG algorithm [39] segregates Adam assign update and magnitude scaling, the RAdam algorithm [40] corrects learning rate variance, the Fromage algorithm [41] controls function space distance, and the AdamW algorithm [42] decouples weight decay from gradient descent.”
Analyzing and Improving the Training Dynamics of Diffusion Models (2023, unknown): “Some optimizers [3, 39, 73] explicitly implement a similar effect by data-dependent re-scaling of the gradient.”
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022, background): “Frobenius Normalization LARS [63], Adafactor [42], Lamb [64], Layca [8], Fromage [6], Nero [28] all involve a normalization step in which the update g (which may be obtained from SGD, Adam, or other optimzers) is normalized to have Frobenius norm equal to that of the parameter w: g ← ‖w‖F ‖g‖F g.”