An empirical investigation of the challenges of real-world reinforcement learning

By Gabriel Dulac-Arnold, Nir Levine, Daniel Jaymin Mankowitz, Jerry Zheng Li, Cosmin Paduraru, Sven Gowal, Todd Hester. 2020

140 citations recorded by the source corpus.

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. In this work, we identify and formalize a series of independent challenges that embody the difficulties that must be addressed for RL to be commonly deployed in real-world systems. For each challenge, we define it formally in the context of a Markov Decision Process, analyze the effects of the challenge on state-of-the-art learning algorithms, and present some existing attempts at tackling it. We believe that an approach that addresses our set of proposed challenges would be readily deployable in a large number of real world problems. Our proposed challenges are implemented in a suite of continuous control environments called realworldrl-suite which we propose an as an open-source benchmark.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, neutral): “[63] Gabriel Dulac-Arnold, Nir Levine, Daniel J.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “, 2013), but also domains that are still challenging for strong online RL algorithms such as real-world RL (RWRL) suite tasks (Dulac-Arnold et al., 2020) and DM Locomotion tasks (Heess et al.”
Inductive biases for deep learning of higher-level cognition (2020, neutral): “This complicates the task of evaluating and comparing learning algorithms but is necessary and we believe, feasible, e.g. see Yu et al. (2017); Packer et al. (2018); Chevalier-Boisvert et al. (2018); Dulac-Arnold et al. (2020); Ahmed et al. (2020).”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, methodology): “With regard to safety, simple RL environments augmented with constraint evaluation (18) and disturbances (5) have been proposed, but lack a unified simulation interface for both safe RL and learning-based control approaches—notably, one that also exposes the available a priori knowledge of a dynamic”
NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning (2021, background): “Other aspects of real-world challenges are also noted in [24].”
Model-Based Offline Planning (2020, methodology): “For RLU the datasets are generated using a 70% performant MPO [10] policy on the original task, and smaller version of the datasets are randomly sampled contiguous episodes [30, 3].”
Deep reinforcement learning for automated stock trading: an ensemble strategy (2020, background): “For future work, it will be interesting to explore more sophisticated model [42], solve empirical challenges [15], deal with large-scale data [7] such as S&P 500 constituent stocks.”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, background): “ronment [74] (ALE), and real-world RL suite [75].”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, methodology): “…the moment of submitting this paper, the following datasets (compatible with RLDS) are in TFDS 7 : a subset of D4RL [15] with tasks from Gym Mujoco [8] and Adroit [37]; the RLUnplugged [19] DMLab [6], Atari [7] and RWRL [13] datasets [19]; two Robosuite [49] datasets generated with the RLDS tools.”
RL Unplugged: A Suite of Benchmarks for Ofﬂine Reinforcement Learning (2020, methodology): “We used either the no challenge setting, which includes unperturbed versions of the tasks, or the easy combined challenge setting (see Section 2.9 of [Dulac-Arnold et al., 2020]), where data logs are generated from an environment that includes effects from combining all the challenges.”