rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch

By Adam Stooke, Pieter Abbeel. 2019

101 citations recorded by the source corpus.

Since the recent advent of deep reinforcement learning for game play and simulated robotic control, a multitude of new algorithms have flourished. Most are model-free algorithms which can be categorized into three families: deep Q-learning, policy gradients, and Q-value policy gradients. These have developed along separate lines of research, such that few, if any, code bases incorporate all three kinds. Yet these algorithms share a great depth of common deep reinforcement learning machinery. We are pleased to share rlpyt, which implements all three algorithm families on top of a shared, optimized infrastructure, in a single repository. It contains modular implementations of many common deep RL algorithms in Python using PyTorch, a leading deep learning library. rlpyt is designed as a high-throughput code base for small- to medium-scale research in deep RL. This white paper summarizes its features, algorithms implemented, and relation to prior work, and concludes with detailed implementation and usage notes. rlpyt is available at this https URL.

Subfields: Artificial Intelligence, Hardware and Architecture, Management Science and Operations Research, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Hardware and Architecture

In Management Science and Operations Research

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, neutral): “Compared to many other frameworks for deep reinforcement learning (Stooke and Abbeel, 2019; Hill et al., 2018; Hoffman et al., 2020), PyMARL is designed with a focus on the multi-agent setting.”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, background): “Additionally, rlpyt [153] implements deep Q-learning, policy gradients and Q-value policy gradients algorithm families on top of a shared, optimized infrastructure, in a single repository in Python. rlpyt is designed as a high throughput code base for small to medium scale research in deep RL.”
Planning from Pixels using Inverse Dynamics Models (2020, methodology): “We built our implementation on top of rlpyt’s (Stooke & Abbeel, 2019) Rainbow implementation (Hessel et al., 2018).”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, background): “Rlpyt (Stooke & Abbeel, 2019) implements highly-efficient asynchronous GPU samplers that share some ideas with our work, although currently it does not include asynchronous policy gradient methods such as IMPALA or APPO.”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, methodology): “Our implementation uses rlpyt (Stooke & Abbeel, 2019) and PyTorch (Paszke et al., 2019).”
Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021, methodology): “Most libraries are targeted at experienced RL researchers, requiring expert knowledge to use (Weng et al., 2020; Hoﬀman et al., 2020; Fujita et al., 2021; Castro et al., 2018; Guadar-rama et al., 2018; Gauci et al., 2018; Stooke and Abbeel, 2019; Kolesnikov, 2018).”
Consistent Dropout for Policy Gradient Reinforcement Learning (2022, background): “…(Liang et al., 2017), OpenAI Baselines (Dhariwal et al., 2017), Tianshou (Weng et al., 2020), Stable-Baselines3 (Raffin et al., 2019), and rlpyt (Stooke & Abbeel, 2019). the policy networks of A2C and PPO agents in three RL frameworks: RLlib (Liang et al., 2017), Stable-Baselines3 (Raffin et…”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “6 The largest improvement is due to utilizing a Double Buffered [40]/Alternating-GPU [52] sampler that allows for 20% more samples to be collected under a given compute budget.”