Diagonal Rescaling For Neural Networks

By Jean Lafond, Nicolas Vasilache, L. Eon Bottou. 2017

12 citations recorded by the source corpus.

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, clarifying popular algorithms such as RMSProp as well as old neural network tricks such as fanin stepsize scaling. The second insight stresses the practical importance of dealing with fast changes of the curvature of the cost.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Large Batch Training of Convolutional Networks (2017, background): “2 ) 5 One can consider LARS as a private case of block-diagonal re-scaling from Lafond et al. (2017).”
Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting (2018, background): “The authors of [16] show how the natural gradient can be used to explain and classify a range of adaptive stepsize strategies for training networks.”
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization (2018, background): “The most common variant of AdaGrad updates an entire vector of per-coefficient stepsizes (Lafond et al., 2017).”
Linear Convergence of Adaptive Stochastic Gradient Descent (2019, background): “(2011) and McMahan and Streeter (2010) update the stepsize on the fly: They either adapt a vector of per-coefficient stepsizes (Kingma and Ba, 2014; Lafond et al., 2017; Reddi et al., 2018a; Shah et al., 2018; Zou et al., 2018b; Staib et al., 2019) or a single stepsize depending on the norm of the g”