What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study

By Anton Raichuk, Piotr Sta'nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L'eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly. 2021

249 citations recorded by the source corpus.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…run times of the algorithms, and make causal attributions of performance gains and transfers of techniques across algorithms difficult, as in the case for many online RL algorithms [Henderson et al., 2017, Tucker et al., 2018, Engstrom et al., 2020, Andrychowicz et al., 2021, Furuta et al., 2021].”
Efficient Online Reinforcement Learning with Offline Data (2023, background): “It is well documented that deep RL algorithms are sensitive to implementation details (Henderson et al., 2018; Engstrom et al., 2020; Andrychowicz et al., 2021; Furuta et al., 2021).”
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, background): “…to reduce the high variance of PG. Engstrom et al. (2020) examine the code level implementation details that are most impact-ful to PPO’s improved performance over TRPO; similarly, Andrychowicz et al. (2021) investigate the effect of PPO’s hyperparameters and provide concrete tuning suggestions.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “Meta analyses of RL algorithms While many papers propose novel algorithms, some recent works focused on meta analyses of some of the popular algorithms, which attracted significant attention due to these algorithms’ high-variance evaluation performances, reproducibility difficulty [9, 28, 65], and f”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, result): “Our work is also closely related to studies on algorithm implementation in the RL community (Engstrom et al., 2020; Andrychowicz et al., 2021; Yu et al., 2022).”
Generalized Proximal Policy Optimization with Sample Reuse (2021, methodology): “[2] both performed extensive empirical analysis on the various implementation choices in these algorithms, while other research has focused on the clipping mechanism used in PPO.”
Understanding R1-Zero-Like Training: A Critical Perspective (2025, unknown): “While advantage normalization is a common trick in RL (Andrychowicz et al., 2021), it is typically computed across an entire batch.”