Value-gradient learning

By Michael Fairbank, Eduardo Alonso. 2012

58 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Learning Continuous Control Policies by Stochastic Value Gradients (2015, methodology): “Following Fairbank [8], we refer to methods that compute the policy gradient through backpropagation as value gradient methods.”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, background): “An alternate approach, based on the idea of backpropagating the gradient of the value function, is developed in (Jordan and Jacobs, 1990; Prokhorov and Wunsch, 1997; Wang and Si, 2001; Hafner and Riedmiller, 2011; Fairbank and Alonso, 2012; Fairbank et al., 2013).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “A class of value gradient methods (Deisenroth & Rasmussen, 2011; Fairbank & Alonso, 2012; Heess et al., 2015) also differentiates through models to train a policy.”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “These approaches have previously been used in control (Werbos, 1992; Fairbank, 2014) and model-based reinforcement learning (Heess et al.”
An Equivalence Between Adaptive Dynamic Programming With a Critic and Backpropagation Through Time (2013, methodology): “An online version is also possible that can be applied to incomplete trajectories [8].”
Sobolev Training for Neural Networks (2017, methodology): “Similar techniques have also been used in critic based Reinforcement Learning (RL), where a critic’s derivatives are trained to match its target’s derivatives [29, 15, 5, 4, 26] using small, sigmoid based models.”
Material for Decoupled Neural Interfaces using Synthetic Gradients (2017, methodology): “These approaches have previously been used in control (Werbos, 1992; Fairbank, 2014) and model-based reinforcement learning (Heess et al.”
Deep Reinforcement Learning (2018, background): “A class of value gradient methods (Deisenroth & Rasmussen, 2011; Fairbank & Alonso, 2012; Heess et al., 2015) also differentiates through models to train a policy.”
Understanding Synthetic Gradients and Decoupled Neural Interfaces (2017, methodology): “These kind of approaches are known in Reinforcement Learning as critic methods (Fairbank, 2014; Heess et al., 2015), but in terms of gradient approximation they do not guarantee any alignment between these signals if only critic is a nonlinear function.”
Expected Policy Gradients for Reinforcement Learning (2018, background): “…policy π parameterised by θ 3: while not converged do 4: g t ← γ t do-integral ( ˆ Q, s, π θ ) g t is the estimated policy gradient as per (10) (Fairbank, 2014; Fairbank and Alonso, 2012; Heess et al., 2015) assume the same parameterisation of the policy as policy gradients, i.e., π is…”