Local Gain Adaptation in Stochastic Gradient Descent

By N. Schraudolph. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods (2013, methodology): “Stochastic meta-descent (Schraudolph, 1999), AdaGrad (Duchi et al., 2010), and SGD-QN (Bordes et al., 2009) rescale the gradient independently for each dimension, and can be viewed as accumulating something similar to a diagonal approximation to the Hessian.”
No more pesky learning rates (2012, methodology): “Stochastic meta-descent (SMD, Schraudolph (1999; 2002)) uses a related multiplicative update of learning rates.”
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates (2019, methodology): “These techniques include using meta-learning procedures to modify the main stochastic algorithm [2, 6, 63, 75, 77, 86, 86], heuristics to adjust the learning rate on the fly [20, 43, 70, 74], and recent adaptive methods inspired by online learning [21, 39, 51, 60, 67, 68, 87].”
Meta-Gradient Reinforcement Learning (2018, background): “There are many works focusing on adapting learning rate through gradient-based methods [Sutton, 1992, Schraudolph, 1999, Maclaurin et al., 2015, Pedregosa, 2016, Franceschi et al., 2017], Bayesian optimisation methods [Snoek et al., 2012], or evolution based hyper-parameter tuning [Jaderberg et…”
Learning to Learn for Global Optimization of Black Box Functions (2016, background): “Several early works approached the problem of meta learning by coupling the model and algorithm into a single system [Sutton, 1992, Schraudolph, 1999].”
Automatic differentiation in machine learning: a survey (2015, methodology): “is to use gain adaptation methods such as stochastic meta-descent (SMD) (Schraudolph, 1999), where stochastic sampling is introduced to avoid local minima and reduce the computational expense.”
La-MAML: Look-ahead Meta Learning for Continual Learning (2020, methodology): “While we adapt the gain based on the correlation between old and new task gradients to make shared progress on all tasks, [4, 25] use the correlation between two successive stochastic gradients to converge faster.”
Online Learning Rate Adaptation with Hypergradient Descent (2017, methodology): “The stochastic meta-descent (SMD) algorithm (Schraudolph et al., 2006; Schraudolph, 1999), developed as an extension of the gain adaptation work by Sutton (1992), operates by multiplicatively adapting local learning rates using a meta-learning rate, employing second-order information from fast Hessi”
SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent (2009, unknown): “Various remedies have been proposed: Stochastic Meta-Descent (Schraudolph, 1999) heuristically determines a learning rate for each coefficient of the parameter vector.”
Learning Rate Adaptation in Stochastic Gradient Descent (2001, background): “Given the inherent efficiency of stochastic gradient descent, various schemes have been proposed recently [1, 19, 20, 21 , 23].”
Accelerated training of conditional random fields with stochastic gradient methods (2006, methodology): “Gain adaptation methods like Stochastic Meta-Descent (SMD) accelerate this process by using second-order information to adapt the gradient step sizes (Schraudolph, 1999, 2002).”
Minimum-Risk Training of Approximate CRF-Based NLP Systems (2012, methodology): “SMD is a second-order method that requires vector-Hessian products.”
Reinforcement Learning: An Introduction (1998, methodology): “…the step-size parameter include RMSprop (Tieleman and Hinton, 2012), Adam (Kingma and Ba, 2015), stochastic meta-descent methods such as Delta-Bar-Delta (Jacobs, 1988), its incremental generalization (Sutton, 1992b, c; Mahmood et al., 2012), and nonlinear generalizations (Schraudolph, 1999, 2002).”
Neural network modeling for near wall turbulent flow (2002, methodology): “The NN training has been performed using an accelerated back-propagation algorithm [22]; the training”
Fast Online Policy Gradient Learning with SMD Gain Vector Adaptation (2005, methodology): “More recent algorithms [3, 4, 10] therefore use multiplicative (hence linear) normalization factors to condition the meta-level update.”
Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming (2006, background): “and signal processing applications in Sutton (1992), Mathews and Xie (1993), Douglas and Mathews (1995), Douglas and Cichocki (1998) and Schraudolph (1999) .”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “Algorithm Algorithm properties What is meta-learned? IDBD, SMD [31, 28] → learning rate SGD(2) [1] ← optimiser RL(2), Meta-RL [9, 40] X recurrent network MAML, REPTILE [11, 24] ← initial params Meta-Gradient [44, 47] → γ, λ, reward Meta-Gradient [39, 45, 41] ← auxiliary tasks, hype”
A Stochastic Quasi-Newton Method for Online Convex Optimization (2007, unknown): “Schraudolph (1999, 2002) further accelerates stochastic gradient descent through online adaptation of a gain vector.”