Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

By Riashat Islam, Peter Henderson, Maziar Gomrokchi, Doina Precup. 2017

282 citations recorded by the source corpus.

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, background): “In (Islam et al. 2017), it is shown that policy network architecture can significantly impact results in both TRPO and DDPG.”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “Overall, this makes it hard to attribute progress in RL and slows down further research [15, 22, 27].”
Benchmarking Model-Based Reinforcement Learning (2019, background): “[24] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup.”
DeepMind Control Suite (2018, background): “Establishing baselines for reinforcement learning problems and algorithms is notoriously difficult (Islam et al., 2017; Henderson et al., 2017).”
Simple random search provides a competitive approach to reinforcement learning (2018, methodology): “Henderson et al. [12] and Islam et al. [13] pointed out that such methodology does not accurately capture the performance of RL methods, which are sensitive to both the choice of random seed and the choice of hyperarameters.”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, background): “[46] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup.”
Expected Policy Gradients (2017, background): “The hyperparameters for DPG and those of EPG that are not related to exploration were taken from an existing benchmark (Islam et al. 2017; Brockman et al. 2016).”
A Tour of Reinforcement Learning: The View from Continuous Control (2018, background): “Henderson et al and Islam et al observed this phenomenon with deep reinforcement learning methods, but our results on linear controllers suggest that such high variability will be a symptom of all model-free methods [35, 36].”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, result): “This paper also provides a guideline for reporting results and comparison against baseline methods [252].”
Deep Reinforcement Learning for Closed-Loop Blood Glucose Control (2020, background): “As performance instability has been noted in other RL domains (Islam et al., 2017; Henderson et al., 2018), this observation is likely relevant to other applications of deep RL in healthcare.”
Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past (2019, methodology): “To make our comparisons fair, analysis meaningful and results reproducible [5, 12, 16], for each variant we use the same SAC code base that we implemented in PyTorch, mainly based on the minimal SAC implementation in [1].”
Leakage and the reproducibility crisis in machine-learning-based science (2023, background): “Besides, publication biases that have been documented in several scientific fields(90,91) can also affect ML-based science.(92,93) [R2] Treat results from ML-based science as tentative.”
On the model-based stochastic value gradient for continuous reinforcement learning (2020, background): “The reproducibility crisis in reinforcement learning (Islam et al., 2017; Henderson et al., 2018; Engstrom et al., 2020; Sinha et al., 2020) makes evaluating and understanding the tradeoﬀs diﬃcult.”
An Introduction to Deep Reinforcement Learning (2018, unknown): “To this end, several works have investigated proper metrics and experimental practices when comparing deep RL algorithms (Henderson et al., 2017b; Islam et al., 2017; Machado et al., 2017b; Whiteson et al., 2011).”
Simple random search of static linear policies is competitive for reinforcement learning (2018, methodology): “[12] pointed out that standard evaluation methodology does not accurately capture the performance of RL methods by showing that existing RL algorithms exhibit high sensitivity to both the choice of random seed and the choice of hyperparameters.”