Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms?

By Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry. 2018

54 citations recorded by the source corpus.

We study how the behavior of deep policy gradient algorithms reflects the conceptual framework motivating their development. We propose a fine-grained analysis of state-of-the-art methods based on key aspects of this framework: gradient estimation, value prediction, optimization landscapes, and trust region enforcement. We find that from this perspective, the behavior of deep policy gradient algorithms often deviates from what their motivating framework would predict. Our analysis suggests first steps towards solidifying the foundations of these algorithms, and in particular indicates that we may need to move beyond the current benchmark-centric evaluation methodology.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

P3O: Policy-on Policy-off Policy Optimization (2019, background): “Further, they tend to have high variance gradient estimates which necessitates a large number of online samples and highly distributed training (Ilyas et al., 2018; Mnih et al., 2016).”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Interestingly, a recent work [151] studying PPO and TRPO arrived at the surprising conclusion that these methods often deviate from what the theoretical framework would predict: gradient estimates are poorly correlated with the true gradient and value networks tend to produce inaccurate predictions ”
Truly Proximal Policy Optimization (2019, background): “This conclusion is partly different from that of Ilyas et al. (2018). They drew a conclusion that “PPO’s performance depends heavily on optimization tricks but not the core clipping mechanism”.”
Understanding the impact of entropy on policy optimization (2018, background): “Recent new work (Ilyas et al., 2018) has shown that in the sample size we have used to visualize the landscape, kinks and bumps are expected but get smoother with larger sample sizes.”
Meta-Q-Learning (2019, methodology): “Policy-gradient-based algorithms versus off-policy methods: Policy-gradient-based methods have high sample complexity (Ilyas et al., 2018).”
Ranking Policy Gradient (2019, background): “For complicated tasks, the value function can align with the relative relationship of action’s return, but the absolute values are hardly accurate (Mnih et al., 2015; Ilyas et al., 2018).”