Policy Gradient Methods for Reinforcement Learning with Function Approximation

By Richard S. Sutton, David A. McAllester, Satinder Singh, Yishay Mansour. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “Therefore, to train the generator, we estimate the gradient with respect to its parameters θ via policy gradients (Sutton et al., 2000).”
Unified Scaling Laws for Routed Language Models (2022, methodology): “To jointly train the experts and the router, we minimize a composite loss formed with the language modelling loss and a policy-gradient term (Sutton et al., 2000) using the selected set of experts as actions.”
Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “The parameters θ of the differentiable policy πθ(at|xt) can be updated using the discounted approximation to the policy gradient (Sutton et al., 2000), which borrowing notation from Schulman et al.”
High-Dimensional Continuous Control Using Generalized Advantage Estimation (2015, methodology): “Simple Monte Carlo estimators of the policy gradient [20, 14] are unbiased and do not require a value function, but their variance scales unfavorably with the time horizon, since the effect of an action is confounded with the effects of past and future actions.”
Learning Continuous Control Policies by Stochastic Value Gradients (2015, background): “Most existing algorithms estimate this gradient in a model-free manner by sampling returns from the real environment and rely on a likelihood ratio estimator [32, 26].”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, methodology): “, 1983; Konda and Tsitsiklis, 2000), where the critic estimates the value of different actions and the actor maximizes rewards by following the policy gradient (Sutton et al., 1999; Peters and Schaal, 2006; Silver et al., 2014).”
Towards Coherent and Cohesive Long-form Text Generation (2018, methodology): “Following Sutton et al. (1999), the gradient of J with respect to θ π is where ρ π is the stationary distribution and Q π ( s, a ) is the expected return from state s and taking action a , both following policy π .”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Table 1: A comparison between Policy Gradients (Sutton et al., 1999) and Distributional Policy Gradients (Parshakova et al.”
Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “A popular class of on-policy algorithms is policy gradient methods (Williams, 1992; Sutton et al., 2000), which have been shown to be effective for a diverse array of complex tasks (Heess et al.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Function approximation to learn a “critic” instead of using rewards directly has been embraced in the RL literature under the name of “actor-critic” methods (see Konda and Tsitsiklis (2000), Sutton et al. (2000), Kakade (2001), Schulman et al. (2015), Mnih et al. (2016), inter alia).”
Adaptive Information Seeking for Open-Domain Question Answering (2021, background): “Even though policies are usually trained via reinforcement learning, reinforcement learning algorithms (Sutton et al., 2000; Mnih et al., 2015) are often”
Learning Parameterized Skills (2012, methodology): “Direct policy search can be performed in three main ways: (1) by using black-box local search methods, such as hill climbing or the NelderMead simplex technique [34]; (2) by using gradient descent methods [86, 6, 72, 9]; or (3) by using EM or crossentropy approaches [31, 46].”
Deep Reinforcement Learning that Matters (2017, neutral): “Here, µ π θ ( s s 0 ) = ∞ t =0 γ t P ( s t = s s 0 ) (Sutton et al. 2000).”
The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, background): “Let V π ( x 0 ) be the value function at some initial state x 0 , the policy gradient theorem (Sutton et al., 2000) says that , where ∇ refers to the gradient w.r.t. policy parameters.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Broadly, these methods can be understood as trading off bias (Sutton et al., 1999; Silver et al., 2014; Lillicrap et al., 2016; Gu et al., 2016) and variance (Precup, 2000; Munos et al., 2016).”