Revisiting Design Choices in Proximal Policy Optimization

By Chloe Ching-Yun Hsu, Celestine Mendler-Dünner, Moritz Hardt. 2020

69 citations recorded by the source corpus.

Proximal Policy Optimization (PPO) is a popular deep policy gradient algorithm. In standard implementations, PPO regularizes policy updates with clipped probability ratios, and parameterizes policies with either continuous Gaussian distributions or discrete Softmax distributions. These design choices are widely accepted, and motivated by empirical performance comparisons on MuJoCo and Atari benchmarks. We revisit these practices outside the regime of current benchmarks, and expose three failure modes of standard PPO. We explain why standard design choices are problematic in these cases, and show that alternative choices of surrogate objectives and policy parameterizations can prevent the failure modes. We hope that our work serves as a reminder that many algorithmic design choices in reinforcement learning are tied to specific simulation environments. We should not implicitly accept these choices as a standard part of a more general algorithm.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, background): “Recent work confirms that PPO’s performance may crucially depend on the choice of policy surrogate objective [11].”
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, background): “Lastly, Hsu et al. (2020) suggest using a soft trust-region penalty and discretizing the action space for PPO to avoid bad local optima in continuous control domains, which is consistent with the improved performance we see in the discrete tasks we consider.”