Challenges of Real-World Reinforcement Learning

By Gabriel Dulac-Arnold, Daniel Jaymin Mankowitz, Todd Hester. 2019

673 citations recorded by the source corpus.

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

A Framework for Data-Driven Robotics (2019, background): “RL: RL has a long history in robotics [14]–[20] but applying RL to robots inherits all the general difficulties of applying RL to the real world [21].”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “Barring a few examples, advancements have not translated quite as well to real-world environments, where it is typically infeasible to try out millions of actions (Dulac-Arnold et al., 2019).”
Critic Regularized Regression (2020, background): “One important reason is that online execution of policies during learning, which we refer to as online RL, is often not feasible or desirable because of cost, safety and ethics [8].”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “This modiﬁcation also brings our domains closer to practice, where partial observability due to ﬂaky sensor readings is common (Dulac-Arnold et al., 2019).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “However, in many practical applications the agent’s ability to continuously act in the environment may be severely limited due to practical concerns (Dulac-Arnold et al., 2019).”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “Dulac-Arnold et al. (2019) and Dulac-Arnold et al. (2020) identify and evaluate respectively a set of 9 challenges that are bottlenecks to implementing RL algorithms, at scale, on applied systems.”
QPLEX: Duplex Dueling Multi-Agent Q-Learning (2020, background): “Recently, offline reinforcement learning has been regarded as a key step for real-world RL applications (Dulac-Arnold et al., 2019; Levine et al., 2020).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “Namely, a common approach to real-world applications (Cabi et al., 2020; Dulac-Arnold et al., 2019; Kalashnikov et al., 2018) is the following iterative training and data-collection paradigm: 1.”
A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, background): “In some applications, the agent’s safety is much more important than the agent’s reward [75].”
Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “The suggestions in Dulac-Arnold et al. (2019) seem straight-forward: warm-starting agents to decrease sample complexity or using inverse reinforcement learning to recover reward functions from demonstrations (Wang et al., 2020) — but they require additional supervision signals that RL was supposed…”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, background): “…RL has shown considerable success in games, robotics, physical system control and computational modeling [Mnih et al., 2015, Silver et al., 2016, Haarnoja et al., 2018, Lazic et al., 2018], large-scale deployment of RL in real-world applications has proven challenging [Dulac-Arnold et al., 2019].”
A review on Deep Reinforcement Learning for Fluid Mechanics (2019, neutral): “[36, 37, 38] for comprehensive domain-agnostic surveys.”
Informed Machine Learning – A Taxonomy and Survey of Integrating Prior Knowledge into Learning Systems (2019, background): “Exploratory actions could hamper user experience [149], [150], so that online reinforcement learning is generally avoided.”
The Benchmark Lottery (2021, methodology): “It includes some widely used domains such as the DM Control Suite [Tassa et al., 2018] and Atari 2600 games [Bellemare et al., 2013], as well as Real-World RL (RWRL) tasks [Dulac-Arnold et al., 2019] and DM Locomotion tasks [Heess et al., 2017].”
Optimizing for the Future in Non-Stationary MDPs (2020, background): “However, real-world applications of reinforcement learning are still lacking (Dulac-Arnold et al., 2019).”
Universal Off-Policy Evaluation (2021, background): “A particular advantage of UnO is the remarkable simplicity with which the estimates and bounds for Fπ or ψ(Fπ) can be extended to account for confounding, distributional shifts, and smooth non-stationarities that are prevalent in real-world applications [30].”
A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems (2022, background): “interesting systems are typically too complex to simulate [79].”