Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta

By Richard S. Sutton. 1992

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Meta-Gradient Reinforcement Learning (2018, methodology): “Our meta-gradient RL approach is based on the principle of online cross-validation [Sutton, 1992], using successive samples of experience.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, a related set of approaches leverage the idea of online cross-validation (Sutton, 1992), which can be deployed in episodic environments leveraging successive episodes for meta-training and meta-testing (Xu, van Hasselt, and Silver, 2018a; Zahavy, Xu, Veeriah, Hessel, Oh, van Hasselt, S”
Learning to Learn for Global Optimization of Black Box Functions (2016, background): “Several early works approached the problem of meta learning by coupling the model and algorithm into a single system [Sutton, 1992, Schraudolph, 1999].”
A Model of Inductive Bias Learning (2000, background): “he information in related tasks to improve average generalization performance across those tasks Singh (1992), Ring (1995), or learning bias from a set of tasks to improve performance on future tasks Sutton (1992), Thrun and Schwartz (1995). 1.2 Overview of the Paper In Section 2 the bias learning m”
Learning values across many orders of magnitude (2016, background): “roblem, such as input normalization. In a different strand of research, several algorithms have been proposed to automatically adapt the step size during learning to handle non-stationarity problems (Sutton, 1992; Mahmood et al., 2012; Schaul et al., 2013). These can also be interpreted as attemptin”
Reinforcement Learning in Situated Agents : Some Theoretical Problems and (1999, methodology): “Sutton (1992) later proposed an incremental ver-sion of the Delta-Bar-Delta rule (IDBD) for use ina linear function approximator.”
Interval Estimation for Reinforcement-Learning Algorithms in Continuous-State Domains (2010, background): “The learning rate, however, can still be adapted based on changes between weight vectors, a more principled approach taken, by the meta-learning algorithm, IDBD [24].”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, methodology): “Additionally, meta-optimizers have also been used for meta-descent [43, 20, 45, 44, 41].”
Learning Rate Adaptation in Stochastic Gradient Descent (2001, background): “Although batch training seems faster for small training sets and networks, on–line training is probably faster for large training sets and BPNNs, it helps escaping local minima and provides a more natural approach for learning non–stationary tasks.”
Evaluating the TD model of classical conditioning (2012, methodology): “These extensions will require new formalisms that may attach additional components to the TD model, such as memory- or model-based learning (Daw, Niv, & Dayan, 2005; Ludvig, Mirian, Sutton, & Kehoe, 2012; Sutton, 1990), step-size adaptation (Pearce & Hall, 1980; Sutton, 1992), or additional configur”
Confidence-weighted linear classification (2008, methodology): “The idea of using parameter-specific variable learning rates has a long history in neural-network learning (Sutton, 1992), although we do not know of a previous model that specifically models confidence in a way that takes into account the frequency of features.”
On Inductive Biases in Deep Reinforcement Learning (2019, methodology): “Instead of tuning the discount manually, we use meta-learning (cf. Sutton, 1992; Bengio, 2000; Finn et al., 2017; Xu et al., 2018) to adapt the discount factor.”
Local Gain Adaptation in Stochastic Gradient Descent (1999, methodology): “where at ≡ ~ w T t ~xt , diagonalizing the Hessian to Ht = diag(~x 2 t ), and adding a positive-bounding operation yields Sutton’s IDBD algorithm [20]:”
Learning to Learn (1998, unknown): “specialized transfer algorithms: Some approaches to transfer have explored how specialized learning methods can be designed to work in a transfer setting [Naik et aI., 1992; Agarwal et aI., 1992; Sutton, 1992].”