Measuring the Reliability of Reinforcement Learning Algorithms
By Stephanie C. Y. Chan, Sam Fishman, John F. Canny, Anoop Korattikara Balan, Sergio Guadarrama. 2019
101 citations recorded by the source corpus.
Inadequate reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a novel set of metrics that quantitatively measure different aspects of reliability. In this work, we address variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. Finally, we apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Anirudh Goyal
- Bernhard Schölkopf
- Yoshua Bengio
- Stefan Bauer
- Filippos Christianos
- Georgios Papoudakis
- Stefano V. Albrecht
- Sergey Levine
In Computational Theory and Mathematics
- Stefano V. Albrecht
- Filippos Christianos
- Yoshua Bengio
- Stefan Bauer
- Anirudh Goyal
- Georgios Papoudakis
- Lukas Schäfer
- Bernhard Schölkopf
In Control and Systems Engineering
- Stefan Bauer
- Bernhard Schölkopf
- Yoshua Bengio
- Anirudh Goyal
- Jan Peters
- Dougal Maclaurin
- Dominik Janzing
- Fabian Pedregosa
Citations in the corpus
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “Chan et al. (2019) distinguishes between several evaluation modes like ”evaluation during training” and ”evaluation after learning”.”
- Comparative Evaluation of Multi-Agent Deep Reinforcement Learning Algorithms (2020, methodology): “[4] and report rankings of all algorithms for the MPE, SMAC and LBF environments in Figure 2 (see [4] for more details on metrics).”
- L2Explorer: A Lifelong Reinforcement Learning Assessment Environment (2022, background): “Despite the groundbreaking success in training autonomous agents, resulting policies tend to be very brittle and generalize poorly (Chan et al. 2019).”