Deep Reinforcement Learning that Matters

By Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. 2017

2,403 citations recorded by the source corpus.

In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

Subfields: Artificial Intelligence, Molecular Biology, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Reinforcement Learning

Citations in the corpus

The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence (2020, background): “Joelle Pineau's points about replicability are also essential (Henderson et al., 2017).”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, positive): “In the quest to draw sound and robust conclusions, we join other recent papers in highlighting the under-acknowledged existence of conclusion replication failure [47, 36, 39, 26, 19].”
Ethical Challenges in Data-Driven Dialogue Systems (2017, background): “Investigations of reproducibility in dialogue systems research can and should comprise singular works, akin to (Henderson et al. 2017).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “…by SAC in our experiments also compare very favorably to results reported by other methods in prior work (Duan et al., 2016; Gu et al., 2016; Henderson et al., 2017), indicating that both the sample efﬁciency and ﬁnal performance of SAC on these benchmark tasks exceeds the state of the art.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “Given the recent concerns in reproducibility (Henderson et al., 2017), we run our experiments across a large number of seeds with fair evaluation metrics, perform ablation studies across each contribution, and open source both our code and learning curves (https://github.”
On Network Design Spaces for Visual Recognition (2019, background): “There has been an encouraging recent trend toward better reproducibility in machine learning [25, 22, 9].”
Show Your Work: Improved Reporting of Experimental Results (2019, methodology): “Lucic et al. (2018) and Henderson et al. (2018) have advocated for using the bootstrap to estimate the mean and variance of the best validation performance.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, methodology): “Finally, best-practice reporting of results should include mean and standard deviation across multiple trials, or at least avoid cherry-picking the best run (Dodge et al., 2020; Henderson et al., 2018).”
Regularized Evolution for Image Classifier Architecture Search (2018, neutral): “Even different implementations of the same algorithm have been shown to produce different results (Henderson et al. 2018).”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, background): “…Learning (RL) algorithms in continuous control domains [65], run extensive evaluations of model-based RL algorithms [66] and on-policy RL methods [67], and shown how Deep RL algorithms can be extremely sensitive to hyperparameter choices and can make reproducing results challenging [68].”
BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018, background): “The challenges for hyperpa-rameter optimization vary substantially across subﬁelds of machine learning; e.g., deep reinforcement learning systems are known to be very noisy (Henderson et al., 2017), while probabilistic deep learning is often very sensitive to a few key hyperparameters.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, positive): “We follow a rigorous evaluation protocol acknowledging the variability in Deep RL (Henderson et al., 2018): each experiment is repeated with 3 seeds (Wu et al.”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “RL methods have been shown to be sensitive to a range of different factors (Henderson et al., 2017b).”
Off-Policy Deep Reinforcement Learning without Exploration (2018, background): “Furthermore, compared to current deep reinforcement learning algorithms, which can require millions of time steps (Duan et al., 2016; Henderson et al., 2017), BCQ attains a high performance in remarkably few iterations.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Henderson et al. [118] described challenges in validating reinforcement learning methods focusing on policy gradient methods for continuous control algorithms such as PPO, DDPG and TRPO as well as in reproducing benchmarks.”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “• Stability, Robustness and Interpretability : DRL algorithms could be relatively unstable, and their performance might deviate a lot between conﬁgurations that only differ slightly from each other (Henderson et al., 2017).”
CoMPS: Continual Meta Policy Search (2021, positive): “This analysis is suggested in Henderson et al. (2017) to provide futher data on the statistical significance of the results.”
Same State, Different Task: Continual Reinforcement Learning without Interference (2021, background): “However, these successes have predominantly focused on learning a single task, with agents often brittle to changes in the dynamics or rewards (or even the seed (Henderson et al. 2018)).”