Reinforcement Learning by Value Gradients
By Michael Fairbank. 2008
30 citations recorded by the source corpus.
The concept of the value-gradient is introduced and developed in the context of reinforcement learning. It is shown that by learning the value-gradients exploration or stochastic behaviour is no longer needed to find locally optimal trajectories. This is the main motivation for using value-gradients, and it is argued that learning value-gradients is the actual objective of any value-function learning algorithm for control problems. It is also argued that learning value-gradients is significantly more efficient than learning just the values, and this argument is supported in experiments by efficiency gains of several orders of magnitude, in several problem domains. Once value-gradients are introduced into learning, several analyses become possible. For example, a surprising equivalence between a value-gradient learning algorithm and a policy-gradient learning algorithm is proven, and this provides a robust convergence proof for control problems using a value function with a general function approximator.
Subfields: Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Reinforcement Learning
- Michael Janner
- Qiyang Li
- Sergey Levine
- Danil V. Prokhorov
- Eduardo Alonso
- Pieter Abbeel
- Michael Fairbank
- Chelsea Finn
Citations in the corpus
- Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et al., 2021).”
- Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “The most obvious is model-based RL, where it is used to train predictive models that can then be used for planning or policy learning [58, 53, 14, 13, 36, 22, 10, 63, 1].”
- An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time (2013, methodology): “We do not consider the Galerkin-based methods (which are also known as residual-gradient methods by [21]) for reasons given by [22] and [7, sec 2.3].”