TensorFlow Agents: Efficient Batched Reinforcement Learning in TensorFlow

By Danijar Hafner, James Davidson, Vincent Vanhoucke. 2017

52 citations recorded by the source corpus.

We introduce TensorFlow Agents, an efficient infrastructure paradigm for building parallel reinforcement learning algorithms in TensorFlow. We simulate multiple environments in parallel, and group them to perform the neural network computation on a batch rather than individual observations. This allows the TensorFlow execution engine to parallelize computation, without the need for manual synchronization. Environments are stepped in separate Python processes to progress them in parallel without interference of the global interpreter lock. As part of this project, we introduce BatchPPO, an efficient implementation of the proximal policy optimization algorithm. By open sourcing TensorFlow Agents, we hope to provide a flexible starting point for future projects that accelerates future research in the field.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Sim-to-Real: Learning Agile Locomotion For Quadruped Robots (2018, background): “Moreover, its computation can be easily parallelized [15].”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, background): “Of note, DQN was originally open-sourced in Lua and has since been re-implemented countless times; similarly, there are a number of publicly available PPO implementations (e.g. Hafner et al., 2017).”
SURREAL: Open-Source Reinforcement Learning Framework and Robot Manipulation Benchmark (2018, methodology): “Prior approaches have relied on asynchronous SGD-style learning (e.g., A3C [27], Gorila [42], ADPG-R [28]), batched data collection for high GPU throughput (e.g., batched A2C [43], GA3C [44], BatchPPO [32]), and more recently, multiple CPU actors for experience generation and single GPU learner for ”
RLlib: Abstractions for Distributed Reinforcement Learning (2017, background): “There are many reinforcement learning libraries (Caspi, 2017; Duan et al., 2016; Hafner et al., 2017; Hesse et al., 2017; Kostrikov, 2017; Schaarschmidt et al., 2017).”
Policy Optimization by Genetic Distillation (2017, methodology): “(2017) propose another objective based on clipping of the likelihood ratio, but we use the adaptive-KL objective due to its better empirical performance (Heess et al., 2017; Hafner et al., 2017).”
Ray RLLib: A Composable and Scalable Reinforcement Learning Library (2017, background): “There are many distributed reinforcement learning libraries [16, 19, 26, 28, 32, 48].”
HybridFlow: A Flexible and Efficient RLHF Framework (2024, unknown): “Gather and concatenate the data from the p=-1, t=0 worker in all DP groups.”
Lyapunov-based Safe Policy Optimization for Continuous Control (2019, methodology): “Second, while CPO is not a backpropagatable algorithm, due to the backtracking line-search procedure and the conjugate gradient iterations for computing natural gradient in TRPO, our algorithms can be trained end-to-end, which is crucial for scalable and efficient implementation (Hafner et al., 2017”