Deep Reinforcement Learning at the Edge of the Statistical Precipice

By Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, Marc G. Bellemare. 2021

950 citations recorded by the source corpus.

Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “Moreover, the Interquartile Mean (IQM) (Agarwal et al., 2021) across all 24 tasks in Figure 1 suggests that APE with InstructGPT outperforms human-engineered prompts, obtaining an IQM of 0.”
Measuring Data (2022, background): “Progress in this vein is beginning (cf. the improved statistical analyses proposed in Agarwal et al. (2021)), but substantial opportunity exists to create a comprehensive notion of measurement in line with those developing across other ML sub-ﬁelds.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “However, as (deep) RL is highly sensitive to variance in the reward function [1, 40], these methods rely on additional models – often doubling the number of learnable parameters – and specialized heuristics to stabilize training.”
Masked Visual Pre-training for Motor Control (2022, methodology): “To reduce randomness in the RL experiments (Agarwal et al., 2021), for each task and model we search for the best learning rate in { 0.0005, 0.001, 0.0015 } with 5 seeds per learning rate (15 runs for each task and model).”
History Compression via Language Models in Reinforcement Learning (2022, methodology): “The performance is evaluated by measuring the interquartile mean (IQM) and 95 % bootstrapped confidence intervals (CIs), as proposed in Agarwal et al. (2021).”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “To aggregate the scores of an agent across games, we use a ﬁxed (usually 20 th ) percentile score (Agarwal et al., 2021).”
Eureka: Human-Level Reward Design via Coding Large Language Models (2023, positive): “That is, whether a reward function is effective is influenced by the particular choice of RL algorithm, and the same reward may perform very differently even under the same optimizer given hyperparameter differences (Henderson et al., 2018; Agarwal et al., 2021).”
AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning (2021, methodology): “Following the recent published work on reliable evaluation for RL (Agarwal et al., 2021b), we utilize 4 evaluation metrics: median, interquartile mean (IQM), mean, MAML (Finnetal.,2017) CAVIA (Zintgrafetal., and optimality gap.”
Transformers are Sample Efficient World Models (2022, methodology): “Interestingly, Schwarzer et al. (2021) note that the median is only inﬂuenced by a few decisive games, as evidenced by the width of the conﬁdence intervals for median scores, even with 100 runs for DrQ, CURL and SPR.”
The Primacy Bias in Deep Reinforcement Learning (2022, methodology): “To provide rigorous evaluations of all algorithms, we follow the guidelines of Agarwal et al. (2021) with the focus on the interquartile mean (IQM) of the performance across tasks.”
Mastering Atari Games with Limited Data (2021, methodology): “[1] propose to use statistical tools to present more robust and efficient aggregate metrics.”
Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “Following Agarwal et al. [2], we summarize these results with the interquartile mean (IQM) of all methods in Figure 4, with bootstrapped 95% conﬁdence intervals constructed from 5k samples per model/env combination.”