Parameter adaptation in stochastic optimization

By Luís B. Almeida, T. Langlois, Jose F. M. Amaral, A. Plakhov. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

No more pesky learning rates (2012, background): “An early diagonal preconditioning schemes was proposed in (Almeida & Langlois, 1999) where the learning rate is adapted as for each problem dimension i , where θ i is gradient of the i th parameter at iteration t , and v i ≈ E ∇ 2 θ i is a recent running average of its square.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Meta-Curvature (2019, background): “In addition, the meta-curvature matrices can be updated online to speed up training deep neural networks (Baydin et al., 2018; Almeida et al., 1999).”
Online Learning Rate Adaptation with Hypergradient Descent (2017, methodology): “We deﬁne the hypergradient descent (HD) method by applying gradient descent on the learning rate of an underlying gradient descent algorithm, independently discovering a technique that has been previously considered in the optimization literature, most notably by Almeida et al. [1].”
3D hand tracking by rapid stochastic gradient descent using a skinning model (2004, background): “which, inserted into (10), gives us the familiar autocorrelation gt ·gt−1 of the gradient, now without the problematic sign function [32, 24].”
Adaptive learning rates and parallelization for stochastic, sparse, non-smooth gradients (2013, background): “There are a number of adaptive settings for SGD learning rates, or equivalently, diagonal preconditioning schemes, to be found in the literature, e.g., [2, 3, 4, 5, 6, 7].”
Fast Online Policy Gradient Learning with SMD Gain Vector Adaptation (2005, methodology): “More recent algorithms [3, 4, 10] therefore use multiplicative (hence linear) normalization factors to condition the meta-level update.”
Value Function Approximation in Reinforcement Learning Using the Fourier Basis (2011, background): “Many online schemes exist for tuning each learning rate, though these result in problem specific learning rates (Aleida et al., 1998).”
Local Gain Adaptation in Stochastic Gradient Descent (1999, methodology): “[19] avoid this pitfall by using a running estimate of the gradient’s stochastic variance as their meta-normalizer (cf.”
Learning Rate Adaptation in Stochastic Gradient Descent (2001, methodology): “Although batch training seems faster for small training sets and networks, on–line training is probably faster for large training sets and BPNNs, it helps escaping local minima and provides a more natural approach for learning non–stationary tasks.”
Gradient Descent: The Ultimate Optimizer (2019, methodology): “Baydin et al. (2017), building on a technique ﬁrst published by Almeida et al. (1999), propose instead up-dating hyperparameters at each step.”