Reinforcement Learning in POMDP's via Direct Gradient Ascent

By Jonathan Baxter, Peter L. Bartlett. 2000

120 citations recorded by the source corpus.

This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm’s chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter 2 [0; 1), which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “When using a parameterized stochastic policy, it is possible to obtain an unbiased estimate of the gradient of the expected total returns (Williams, 1992; Sutton et al., 1999; Baxter & Bartlett, 2000); these noisy gradient estimates can be used in a stochastic gradient ascent algorithm.”
Multiagent learning using a variable learning rate (2002, background): “There are single-agent learning techniques that are capable of playing stochastic policies [1,2,16,30].”
Model-based contextual policy search for data-efficient generalization of robot skills (2017, background): “The context is varied from x ∈ [5, 15]m and y ∈ [0, 3]m.”
Point-based value iteration: an anytime algorithm for POMDPs (2003, methodology): “Gradient-based policy search methods have also been used to optimize POMDP solutions [Baxter and Bartlett, 2000; Kearns et al. , 1999; Ng and Jordan, 2000], successfully solving multi-dimensional, continuous-state problems.”
Exploring parameter space in reinforcement learning (2010, background): “Various approaches have been proposed to reduce this variance [1, 3, 14, 26].”
Parameter-exploring policy gradients (2010, unknown): “Various approaches have been proposed to reduce this variance Baxter and Bartlett (2000); Aberdeen (2003); Peters and Schaal (2006); Sutton et al. (2000).”
Simultaneous Adversarial Multi-Robot Learning (2003, methodology): “It would be interesting to combine WoLF with other policy gradient techniques, such as [Williams, 1992; Baxter and Bartlett, 2000].”
Metalearning and neuromodulation (2002, background): “Although the discount factor γ has be set large enough to achieve good behaviors over the long run, the closer γ approaches one, the more diﬃcult it is to reliably predict the corresponding future reward (Littman et al., 1995; Baxter & Bartlett, 2000).”
Policy Gradients with Parameter-Based Exploration for Control (2008, background): “Various approaches have been proposed to reduce this variance [6,7,2,8].”
State-Dependent Exploration for Policy Gradient Methods (2008, methodology): “2 GPOMDP [10], also known as the Policy Gradient Theorem [6], does consider single step rewards.”
Coordinated Reinforcement Learning (2002, background): “The shortcomings of value function based methods have led to a surge of interest in direct policy search methods [13; 12; 1; 15; 9].”