On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift

By Alekh Agarwal, Sham M. Kakade, Jason D. Lee, Gaurav Mahajan. 2019

663 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, methodology): “…those from deterministic expert policies, are particularly problematic for offline RL algorithms and may cause divergence both empirically [Fujimoto et al., 2018, Kumar et al., 2019] and theoretically [Munos, 2003, Farahmand et al., 2010, Kumar et al., 2019, Agarwal et al., 2019a, Du et al., 2020].”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “The uniform sampling of actions in D follows similar settings in related work [5] and in principle can be replaced with any distribution uniformly bounded from below by η > 0 and scaling our derived bounds by 1 |A|η .”
The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, unknown): “We learn from (Agarwal et al., 2021) that, when we are updating policy using natural policy gradient via gradient backtracking, z k +1 s,a − z ks,a satisfies,”
Bellman-consistent Pessimism for Offline Reinforcement Learning (2021, methodology): “This property has been leveraged in many prior works, although typically in online RL settings [Even-Dar et al., 2009; Agarwal et al., 2019; Geist et al., 2019; Cai et al., 2020; Shani et al., 2020].”
Provably Efficient Exploration in Policy Optimization (2019, methodology): “…on the aforementioned line of recent work (Fazel et al., 2018; Yang et al., 2019a; Abbasi-Yadkori et al., 2019a;b; Bhandari & Russo, 2019; Liu et al., 2019; Agarwal et al., 2019; Wang et al., 2019) on the computational efficiency of policy optimization, which covers PG, NPG, TRPO, PPO, and AC.”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “The analysis techniques are most similar to the policy gradient literature since they also consider the soft-max policies (Agarwal et al., 2021; Cai et al., 2020; Zanette et al., 2021b; Zhong & Zhang, 2023).”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “Our data augmentation approach in Section 3 relates to exploration requirements for optimal learning via policy gradient methods (Agarwal et al., 2021), learning recovery policies (Thananjeyan et al., 2021), and safe control theory (Burridge et al., 1999; Ames et al., 2019).”
Reward-Free Exploration for Reinforcement Learning (2020, methodology): “The η here is essential η/(1− γ) in (Agarwal et al., 2019).”
Representation Learning for Online and Offline RL in Low-rank MDPs (2021, methodology): “More specifically, we propose an algorithm REP-LCB—Lower Confidence Bound driven Reprepresentation Learning for offline RL, that given an offline dataset, can learn to compete against any policy (including history-dependent policies) as long as it is covered by the offline data where the coverage is”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, unknown): “DCRL [222] On-line Learning [6] / Actor-Critic”
Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning? (2019, methodology): “Second, more recently, Agarwal et al. (2019) showed that policy gradient styles of algorithms also have comparable guarantees.”