Implementation Matters in Deep RL: A Case Study on PPO and TRPO

By Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, Aleksander Madry. 2020

322 citations recorded by the source corpus.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, positive): “Our empirical analysis of PPO’s implementation and hyperparameter factors in multi-agent settings is similar to the studies of policy-gradient methods in single-agent RL [33, 17, 9, 1].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…run times of the algorithms, and make causal attributions of performance gains and transfers of techniques across algorithms difficult, as in the case for many online RL algorithms [Henderson et al., 2017, Tucker et al., 2018, Engstrom et al., 2020, Andrychowicz et al., 2021, Furuta et al., 2021].”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, methodology): “See [27] for the description of this technique.”
Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, background): “Implementation details in Deep RL algorithms significantly affect their achieved returns [Engstrom et al., 2019].”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, result): “Our work is also closely related to studies on algorithm implementation in the RL community (Engstrom et al., 2020; Andrychowicz et al., 2021; Yu et al., 2022).”
CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms (2022, background): “However, understanding these implementation details is essential because they could signiﬁcantly aﬀect performance (Engstrom et al., 2020).”
Generalized Proximal Policy Optimization with Sample Reuse (2021, background): “To see why this is true, note that the clipping mechanism has no impact at the beginning of each policy update since each probability ratio begins at the center of the clipping range [5].”
Revisiting Design Choices in Proximal Policy Optimization (2020, unknown): “Recent ablations studies [3, 10] investigate how some hyperparameters and design choices a ﬀ ect the performance of PPO and other on-policy RL algorithms on MuJoCo benchmarks.”
Comparative Evaluation of Multi-Agent Deep Reinforcement Learning Algorithms (2020, methodology): “Given the sensitivity of RL approaches to hyperparameters, seeds [14] and implementation details [10], we also evaluate the reliability of MARL approaches on each environment.”
Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021, background): “A major challenge is that small implementation details can have a substantial eﬀect on performance – often greater than the diﬀerence between algorithms (Engstrom et al., 2020).”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “RL and RLHF have many subtle implementation details that can significantly impact training stability [Engstrom et al., 2020, Huang et al., 2022].”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “It is a well known problem that RL algorithms are sensitive to small adjustments (Henderson et al., 2017; Engstrom et al., 2019).”
AdaLead: A simple and robust adaptive greedy search algorithm for sequence design (2020, background): “As mentioned in the main text, TRPO algorithms can be sensitive to implementation (Engstrom et al., 2019).”
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, background): “…Ilyas et al. (2020) analyze the accuracy of the estimated gradients and suggest the usage of large batch sizes to reduce the high variance of PG. Engstrom et al. (2020) examine the code level implementation details that are most impact-ful to PPO’s improved performance over TRPO; similarly,…”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “[10] focus solely on PPO and TRPO, two on-policy algorithms, and discuss (2)We here regard as co-adaptation the indispensable implementation or code decisions that do not stem directly from the conceptual algorithmic development, but from empirical considerations.”
Efficient Online Reinforcement Learning with Offline Data (2023, background): “It is well documented that deep RL algorithms are sensitive to implementation details (Henderson et al., 2018; Engstrom et al., 2020; Andrychowicz et al., 2021; Furuta et al., 2021).”
What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021, background): “The most closely related work to ours is probably Engstrom et al. [27] where the authors investigate code-level improvements in the PPO [17] code base and conclude that they are responsible for the most of the performance difference between PPO and TRPO [10].”