What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

By Marcin Andrychowicz, Anton Raichuk, Piotr Sta'nczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, L'eonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski and 2 more. 2020

283 citations recorded by the source corpus.

In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

Subfields: Artificial Intelligence, Computer Science Applications, Information Systems, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Olivier Pietquin
5. Danijar Hafner
6. Ruslan Salakhutdinov
7. Yoshua Bengio
8. Timothy P. Lillicrap

In Computer Science Applications

1. L'eonard Hussenot
2. Lukas Schäfer
3. Olivier Pietquin
4. Matthieu Geist
5. Stefano V. Albrecht
6. Georgios Papoudakis
7. Filippos Christianos
8. Jacob Steinhardt

In Information Systems

1. Kush Bhatia
2. Olivier Pietquin
3. Matthieu Geist
4. Stefano V. Albrecht
5. Marcin Andrychowicz
6. Jimmy Ba
7. Timothy P. Lillicrap
8. Jacob Steinhardt

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Jimmy Ba
4. Danijar Hafner
5. Olivier Pietquin
6. Timothy P. Lillicrap
7. Chelsea Finn
8. J. Pašukonis

In Evaluation & Benchmarks

1. L'eonard Hussenot
2. Anton Raichuk
3. Olivier Pietquin
4. Matthieu Geist
5. Manu Orsini
6. Olivier Bachem
7. Stefano V. Albrecht
8. Marcin Andrychowicz

Citations in the corpus

• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Model capacity is only one proxy for agent capabilities, and larger models do not always lead to more capable agents (Andrychowicz et al., 2020).”
• Hyperparameter Selection for Imitation Learning (2021, background): “Indeed, RL algorithms are known to be very sensitive to the values of their numerous HPs (Henderson et al., 2018; Andrychowicz et al., 2020).”
• Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks (2020, background): “Implementation details in reinforcement learning significantly affect the returns that each algorithm achieves [Andrychowicz et al., 2021].”
• Revisiting Design Choices in Proximal Policy Optimization (2020, unknown): “For example, in a recent large-scale study [3] of more than 50 design choices in on-policy RL algorithms, the choices of clipping as regularization and Gaussian policy are not included in the studied design choices.”
• ReFT: Reasoning with Reinforced Fine-Tuning (2024, methodology): “Finally, we also experiment with a value model that has no parameter shared with the policy model (Andrychowicz et al., 2021; Cobbe et al., 2021b).”
• Mastering Diverse Domains through World Models (2023, methodology): “However, applying reinforcement learning algorithms to sufficiently new tasks—such as moving from video games to robotics tasks— requires substantial effort, expertise, and computational resources for tweaking the hyperparameters of the algorithm 13 .”