Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO

By Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry. 2020

333 citations recorded by the source corpus.

We study the roots of algorithmic progress in deep policy gradient algorithms through a case study on two popular algorithms: Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO). Specifically, we investigate the consequences of "code-level optimizations:" algorithm augmentations found only in implementations or described as auxiliary details to the core algorithm. Seemingly of secondary importance, such optimizations turn out to have a major impact on agent behavior. Our results show that they (a) are responsible for most of PPO's gain in cumulative reward over TRPO, and (b) fundamentally change how RL methods function. These insights show the difficulty and importance of attributing performance gains in deep reinforcement learning. Code for reproducing our results is available at this https URL .

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? (2020, methodology): “In addition, code-level design choices have been shown to signiﬁcantly impact PPO performance in practice [8].”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, methodology): “Our vanilla PPO agents achieve similar or better performance than reported in the literature [14, 25, 22].”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “The reproducibility crisis in reinforcement learning (Islam et al., 2017; Henderson et al., 2018; Engstrom et al., 2020; Sinha et al., 2020) makes evaluating and understanding the tradeoﬀs diﬃcult.”
RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, background): “Although PPO is a well-established DRL method with numerous studies showcasing its effectiveness [34, 11], it learns in a trial-and-error fashion by interacting with the environment and is generally significantly less stable and less efficient as compared to supervised learning [7].”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, background): “However, getting the PPO work is challenging in the context of LLMs [Choshen et al., 2019; Engstrom et al., 2020].”
Detecting and Preventing Hallucinations in Large Vision Language Models (2023, background): “At the same time, it avoids the unstable and brittle process of training models with RL[7].”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “Reward clipping is widely adopted by previous works like [Mnih et al., 2015, Engstrom et al., 2020] as well as the Deepspeed RLHF implementation.”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “…et al., 2019; Wu et al., 2021a; Ouyang et al., 2022; Bai et al., 2022a) is to use the standard deep RL algorithms like PPO to optimize the regularized reward: However, PPO is significantly less stable and sensitive to implementation as compared to SFT (Choshen et al., 2019; Engstrom et al., 2020).”
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework (2024, positive): “Furthermore, OpenRLHF applies several tricks to stabilize training in the PPO implementation [8] including • Predict reward only on the end-of-text token of the sequence.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “According to [39], deep policy-gradient algorithm Proximal Policy Optimization (PPO) [135] contains several reﬁnements, such as improved initialization methods and reward scaling and clipping, over its predecessor, Trusted Region Policy Optimization (TRPO) [134].”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, positive): “For reward optimization, PPO has been shown to be effective in various studies [31, 12].”
Experience Replay with Likelihood-free Importance Weights (2020, methodology): “Our implementations are all based on RLkit † which gives a fair comparison between the baselines and our approach given the difﬁculty to reproduce the best RL results [29, 10].”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “While effective, this pipeline is often criticized for its complexity, training instability, and sensitivity to hyperparameter tuning (Engstrom et al., 2020; Azar et al., 2024; Casper et al., 2023; Rafailov et al., 2023).”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “However, it is known that even in the best case, tuning the DRL method to its best performance requires extensive efforts in hyper-parameter selection and code-level optimization (Choshen et al., 2019; Engstrom et al., 2020).”