Ray RLLib: A Composable and Scalable Reinforcement Learning Library

By Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph E. Gonzalez, Ken Goldberg, Ion Stoica. 2017

181 citations recorded by the source corpus.

Reinforcement learning (RL) algorithms involve the deep nesting of distinct components, where each component typically exhibits opportunities for distributed computation. Current RL libraries offer parallelism at the level of the entire program, coupling all the components together and making existing implementations difficult to extend, combine, and reuse. We argue for building composable RL components by encapsulating parallelism and resource requirements within individual components, which can be achieved by building on top of a flexible task-based programming model. We demonstrate this principle by building Ray RLLib on top of Ray and show that we can implement a wide range of state-of-the-art algorithms by composing and reusing a handful of standard components. This composability does not come at the cost of performance --- in our experiments, RLLib matches or exceeds the performance of highly optimized reference implementations. Ray RLLib is available as part of Ray at this https URL

Subfields: Artificial Intelligence, Mechanical Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

Citations in the corpus

rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, background): “Rllib [31], built on top of Ray [32], focuses on distributed computing, possibly complicating small experiments.”
Horizon: Facebook's Open Source Applied Reinforcement Learning Platform (2018, background): “Currently, several platforms have been developed that address different parts of this end-to-end applied RL challenge [2, 4, 10, 29], however to our knowledge, no single system offers an end-toend solution.”
Deep Reinforcement Learning: An Overview (2017, methodology): “ (ALE) (Bellemare et al.,2013;Machado et al.,2017) is a framework composed of Atari 2600 games to develop and evaluate AI agents. Ray RLlib: A Composable and Scalable Reinforcement Learning Library (Liang et al., 2017c),http://ray.readthedocs.io/en/latest/rllib.html OpenAI Gym (https://gym.openai.co”
GPU-Accelerated Robotic Simulation for Distributed Reinforcement Learning (2018, methodology): “In their benchmarks, the authors were able to scale ES with Ray RLLib to more than 8000 CPU cores, learning the humanoid walking task in just 3.7 minutes.”
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search (2020, methodology): “amp; Hutter, 2019). One solution to guard against these problems would be a common library of NAS methods that provides primitives to construct different algorithm variants, similar to what as RLlib (Liang et al., 2017) offers for the ﬁeld of reinforcement learning. Our paper makes a ﬁrst step into ”
Lessons from Real-World Reinforcement Learning in a Customer Support Bot (2019, background): “The original motivation of RLlib was to perform RL in a simulated environment with very easy to use abstractions that can seamlessly scale the computation without the user having to worry about details of distributed systems.”
Flow: A Modular Learning Framework for Mixed Autonomy Traffic (2017, methodology): “The implementation builds upon SUMO ( S imulation of U rban MO bility) [102] for trafﬁc modeling, Ray RLlib [103] for RL methods, and OpenAI gym [101] for the MDP interface.”
A review of cooperative multi-agent deep reinforcement learning (2019, background): “Similarly, Ray platform [96] also recently started to support multi-agent environments and some of the known algorithms are also added in rllib repository [78, 77].”
Benchmarks for reinforcement learning in mixed-autonomy traffic (2018, methodology): “The version of RLlib used to run the code is available at https: //github.com/eugenevinitsky/ray at commit “0f45f80”.”
Consistent Dropout for Policy Gradient Reinforcement Learning (2022, methodology): “Similarly, RLlib experienced instability in five of six A2C runs and two of six PPO runs.”