Benchmarking Reinforcement Learning Algorithms on Real-World Robots

By A. Mahmood, Dmytro Korenkevych, Gautham Vasan, Will W. K. Ma, James Bergstra. 2018

189 citations recorded by the source corpus.

Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

BADGR: An Autonomous Self-Supervised Learning-Based Navigation System (2020, background): “…learning navigation policies directly from real-world experience, but typically require a person [18], [21] or SLAM algorithm [22] to gather the data, assume access to the ground-truth robot state [23], learn using low-bandwidth sensors [24], or only perform collision avoidance [4], [25].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “Mahmood et al. (2018) provide benchmarks using four off-the-shelf RL algorithms and evaluate the performance on multiple commercially available robots.”
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, background): “, 2017), and real robot systems (Mahmood et al., 2018), off-policy methods currently are state-of-the-art in terms of sample complexity.”
Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning (2021, background): “…with two typical embodi-ments of Trust Region Policy Optimisation (TRPO) (Schulman et al., 2015a) and Proximal Policy Optimisation (PPO) (Schulman et al., 2017) algorithms, offer supreme empirical performance in both discrete and continuous RL problems (Duan et al., 2016; Mahmood et al., 2018).”
Environments for Lifelong Reinforcement Learning (2018, methodology): “Aligned with recent efforts toward reproducible research, [Mahmood et al., 2018] introduced benchmark tasks for physical robots which allow experiments to be reproduced in different locations and under diverse conditions.”
An Exploration of Embodied Visual Exploration (2020, background): “In the spirit of recent influential studies in other domains (Duan et al. 2016; Mahmood et al. 2018; Goyal et al. 2019; Mishkin et al. 2019; Savva et al. 2019), we aim first and foremost to provide a reliable, unbiased, and thorough view of the state of the art that can be a reference to the field m”
How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? (2019, background): “While there are promising successes in applying RL algorithms directly on real systems ([7], [8], [9], [10], [11]), their performance on more complex systems remains bottle-necked by the relative data inefﬁciency of RL algorithms.”