Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments

By Kaleigh Clary, Emma Tosch, John Foley, David D. Jensen. 2019

27 citations recorded by the source corpus.

Reproducibility in reinforcement learning is challenging: uncontrolled stochasticity from many sources, such as the learning algorithm, the learned policy, and the environment itself have led researchers to report the performance of learned agents using aggregate metrics of performance over multiple random seeds for a single environment. Unfortunately, there are still pernicious sources of variability in reinforcement learning agents that make reporting common summary statistics an unsound metric for performance. Our experiments demonstrate the variability of common agents used in the popular OpenAI Baselines repository. We make the case for reporting post-training agent performance as a distribution, rather than a point estimate.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Luke S. Zettlemoyer
3. Percy Liang
4. Quoc V. Le
5. Mohit Bansal
6. Jianfeng Gao
7. Yoshua Bengio
8. Yejin Choi

In Management Science and Operations Research

1. Pablo Samuel Castro
2. Kaleigh Clary
3. David D. Jensen
4. Rishabh Agarwal
5. Aaron C. Courville
6. Max Schwarzer
7. T. Henighan
8. Amanda Askell

In Reinforcement Learning

1. Rishabh Agarwal
2. Pablo Samuel Castro
3. Aaron C. Courville
4. Max Schwarzer
5. Marc G. Bellemare
6. Sergey Levine
7. Dario Amodei
8. David D. Jensen

Citations in the corpus

• Predictability and Surprise in Large Generative Models (2022, background): “For example, [17] documents strong run-to-run irreproducibility in reinforcement learning on Atari games when only changing the initial random seed.”
• ToyBox: Better Atari Environments for Testing Reinforcement Learning Agents (2018, background): “Although outside the scope of this work, there is a recent growing interest in the role of variability during evaluation in machine learning, as well as RL spcifically [6, 5, 11].”
• Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[17] Kaleigh Clary, Emma Tosch, John Foley, and David Jensen.”
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, methodology): “Due to issues with variability across agents and environments (Henderson et al., 2017; Clary et al., 2018; Jordan et al., 2018), we trained ten replicates for each of these training algorithms, differentiated by their random seed.”