TorchBeast: A PyTorch Platform for Distributed RL

By Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktaschel, Edward Grefenstette. 2019

62 citations recorded by the source corpus.

TorchBeast is a platform for reinforcement learning (RL) research in PyTorch. It implements a version of the popular IMPALA algorithm for fast, asynchronous, parallel training of RL agents. Additionally, TorchBeast has simplicity as an explicit design goal: We provide both a pure-Python implementation ("MonoBeast") as well as a multi-machine high-performance version ("PolyBeast"). In the latter, parts of the implementation are written in C++, but all parts pertaining to machine learning are kept in simple Python using PyTorch, with the environments provided using the OpenAI Gym interface. This enables researchers to conduct scalable RL research using TorchBeast without any programming knowledge beyond Python and PyTorch. In this paper, we describe the TorchBeast design principles and implementation and demonstrate that it performs on-par with IMPALA on Atari. TorchBeast is released as an open-source package under the Apache 2.0 license and is available at \url{this https URL}.

Subfields: Hardware and Architecture, Mechanical Engineering, Software, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Hardware and Architecture

1. Gabriel Barth-Maron
2. Nando de Freitas
3. Edward Grefenstette
4. Alexander Novikov
5. Ziyun Wang
6. Fan Yang
7. Jacob Steinhardt

In Mechanical Engineering

1. Serkan Cabi
2. Gabriel Barth-Maron
3. Abhinav Kumar Gupta
4. Nando de Freitas
5. Caglar Gulcehre
6. Sergio Gomez Colmenarejo
7. Alexander Novikov
8. Ziyun Wang

In Software

1. Noah D. Goodman
2. Ziyun Wang
3. Tim Rocktaschel
4. T. Paine
5. Jacob Steinhardt

In Reinforcement Learning

1. Sergey Levine
2. Ziyun Wang
3. Pieter Abbeel
4. Nando de Freitas
5. T. Paine
6. Roozbeh Mottaghi
7. Abhinav Kumar Gupta
8. Alexander Novikov

Citations in the corpus

• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, methodology): “…to optimize policies for the traffic and COVID environments, SAC (Haarnoja et al., 2018) to optimize the policies for the glucose environment, and torch-beast (K¨uttler et al., 2019), a PyTorch implementation of IMPALA (Espeholt et al., 2018), to optimize the policies for the Atari environment.”
• Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Similarly, TorchBeast (Küttler et al., 2019) is another IMPALA implementation written in Torch.”
• The NetHack Learning Environment (2020, methodology): “For the main experiments, we train the agent’s policy for 1B steps in the environment using IM-PALA [24] as implemented in TorchBeast [44].”
• Improving Intrinsic Exploration with Language Abstractions (2022, methodology): “We evaluate L-AMIGo, AMIGo, L-NovelD, and NovelD, implemented in the TorchBeast [27] implementation of IMPALA [17], a common asynchronous actor-critic method.”
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “This may be because we use the TorchBeast (Küttler et al., 2019) implementation of IMPALA, while the results in Rolnick et al. (2019) and Schwarz et al. (2018b) use an earlier, pre-release version of IMPALA.”