Direct gradient-based reinforcement learning

By Jonathan Baxter, Peter L. Bartlett. 2000

133 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Learning Parameterized Skills (2012, methodology): “Direct policy search can be performed in three main ways: (1) by using black-box local search methods, such as hill climbing or the NelderMead simplex technique [34]; (2) by using gradient descent methods [86, 6, 72, 9]; or (3) by using EM or crossentropy approaches [31, 46].”
Decoupled Neural Interfaces using Synthetic Gradients (2016, methodology): “Our synthetic gradient model is most analogous to a value function which is used for gradient ascent (Bax- ter & Bartlett, 2000) or critics for training neural networks (Schmidhuber, 1990).”
Deep learning in neural networks: An overview (2014, methodology): “usal chains of modiﬁable parameters—it neither cares for their existence, nor tries to exploit them. An important class of DS methods for NNs are Policy Gradient methods (Williams, 1986, 1988, 1992a; Baxter and Bartlett, 1999;Sutton et al., 1999a;Aberdeen,2003;Ghavamzadehand Mahadevan,2003;Kohl and ”
Policy Tree: Adaptive Representation for Policy Gradient (2015, methodology): “We used the GPOMDP/Policy Gradient algorithm with optimal baseline (Peters and Schaal 2006) to measure the policy gradient in all of the algorithms.”
Solving Deep Memory POMDPs with Recurrent Policy Gradients (2007, methodology): “, the episode length can be truncated or we can use a discount factor γ̂ which is lower than the true γ as in the GPOMDP algorithm [18] where 0 < γ̂ < γ.”
Learning and Transferring Roles in Multi-Agent Reinforcement (2008, methodology): “Each agents behavior is optimized using OLPOMDP an online policy gradient optimization technique (Baxter and Bartlett 1999) using the sampled policy as an initial starting point.”
Policy Gradient Methods for Robotics (2006, background): “However, in order to clarify the relationship to [17], [25], we note that the term ∑H l=kalrl in the policy gradient theorem is equivalent to a monte-carlo estimate of the value func-”
On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient (2010, methodology): “GPOMDP (GP) and IS GPOMDP (ISGP) are also plotted for reference purposes. three components, which is equivalent to Peshkin and Shelton's algorithm [10], which we label PS.”
Optimizing Average Reward Using Discounted Rewards (2001, methodology): “by [ 2 ], extensions include modifying the algorithm to compute the Hessian in a Partially Observable Markov Decision Process and in continuous state, action, and control spaces.”
Reinforcement Learning by Value Gradients (2008, background): “This problem has led to a major alternative RL approach which works without a value function at all, i.e. policy-gradient learning (PGL) algorithms (Williams, 1992; Baxter and Bartlett, 2000; Werbos, 1990), which do have the desired convergence guarantees.”
Provably Efficient Exploration in Policy Optimization (2019, background): “, 2019) answers the computational question affirmatively by proving that a wide variety of policy optimization algorithms, such as policy gradient (PG) (Williams, 1992; Baxter and Bartlett, 2000; Sutton et al., 2000), natural policy gradient (NPG) (Kakade, 2002), trust-region policy optimization (TR”
Bayesian Reinforcement Learning in Factored POMDPs (2018, methodology): “More traditional approaches including the U-Tree algorithm [29] (and its modifications), EM-based algorithms such as [25] and policy gradient methods [4], typically do not suffer from the same lack of sample efficiency.”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, background): “There is a huge body of literature on single-agent actor-critic algorithms, which are based on the policy gradient theorem (Baxter and Bartlett, 2000).”
Model-Free reinforcement learning with continuous action in practice (2012, unknown): “AC without a critic (αv = 0), which we call A, is similar to REINFORCE and to OLPOMDP [15].”