Acme: A Research Framework for Distributed Reinforcement Learning

By Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal M. P. Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli and 10 more. 2020

248 citations recorded by the source corpus.

Deep reinforcement learning has led to many recent-and groundbreaking-advancements. However, these advances have often come at the cost of both the scale and complexity of the underlying RL algorithms. Increases in complexity have in turn made it more difficult for researchers to reproduce published RL algorithms or rapidly prototype ideas. To address this, we introduce Acme, a tool to simplify the development of novel RL algorithms that is specifically designed to enable simple agent implementations that can be run at various scales of execution. Our aim is also to make the results of various RL algorithms developed in academia and industrial labs easier to reproduce and extend. To this end we are releasing baseline implementations of various algorithms, created using our framework. In this work we introduce the major design decisions behind Acme and show how these are used to construct these baselines. We also experiment with these agents at different scales of both complexity and computation-including distributed versions. Ultimately, we show that the design decisions behind Acme lead to agents that can be scaled both up and down and that, for the most part, greater levels of parallelization result in agents with equivalent performance, just faster.

Subfields: Artificial Intelligence, Mechanical Engineering, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Mechanical Engineering

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, neutral): “Compared to many other frameworks for deep reinforcement learning (Stooke and Abbeel, 2019; Hill et al., 2018; Hoffman et al., 2020), PyMARL is designed with a focus on the multi-agent setting.”
Critic Regularized Regression (2020, methodology): “D4PG implemented in Acme [14], following their network architectures and hyper-parameters.”
Hyperparameter Selection for Imitation Learning (2021, methodology): “We implemented our algorithms in the Acme framework (Hoffman et al., 2020) using JAX (Bradbury et al., 2018) for automatic differentiation and Flax (Heek et al., 2020) for neural networks computation.”
Benchmarks for Deep Off-Policy Evaluation (2021, methodology): “All ofﬂine RL algorithms are implemented using the Acme framework (Hoffman et al., 2020).”
Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “Our implementations are based on open-sourced D4PG implementation from Acme (Hoﬀman et al., 2020).”
Primal Wasserstein Imitation Learning (2020, methodology): “In this section, we present the implementation of PWIL and perform an empirical evaluation, based on the ACME framework (Hoffman et al., 2020).”
Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation (2021, methodology): “All environments were evaluated with the same standard implementation of SAC[28] with environments evaluated on CPU and learning on a 2x2 TPUv2—i.”
Reverb: A Framework For Experience Replay (2021, methodology): “To accelerate data collection, many RL experiments split the agent into two distinct parts: data generators ( actors ) and data consumers ( learners ), which are run in parallel (Horgan et al., 2018; Hoffman et al., 2020).”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “We note that it is possible to adapt DrQ-v2 to a distributed setup, as has been done for DDPG in prior work [Barth-Maron et al., 2018, Hoffman et al., 2020].”
AndroidEnv: A Reinforcement Learning Platform for Android (2021, methodology): “For our experiments, we used the Acme framework (Hoffman et al., 2020) and its TensorFlow (Abadi et al., 2015) agents available at Acme’s Github Repository.2 Since the action interface in AndroidEnv is a hybrid of discrete and continuous components, we defined some Wrappers (described below) for…”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, methodology): “We chose these two algorithms for benchmarking performance as they (1) yield SOTA performance on the dm-control suite (see e.g., Hoffman et al. 2020; Barth-Maron et al. 2018; 2) they are both fundamentally different algorithms (DMPO is an EM-style policy iteration algorithm with a stochastic policy…”
RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning (2021, methodology): “An additional set of examples is released as part of a Notebook 9 , and performance best practices are available as a separate Notebook Besides, there are two agents already released [23] that make use of RLDS datasets.”
Mastering diverse control tasks through world models (2025, unknown): “We use the Acme implementations 51 of these algorithms, use the same IMPALA network we used for PPO, and tune the learning rate and entropy regularizers.”
What Matters for Adversarial Imitation Learning? (2021, methodology): “We release this generic AIL agent, implemented in JAX [25] as part of the Acme [49] framework: https://github.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “We reproduce all algorithms based on pytorch RL library [14], referring their original implementations [22, 48, 42, 27].”
Gymnasium: A Standard Interface for Reinforcement Learning Environments (2025, unknown): “Several alternative libraries offer different approaches to defining and interacting with RL environments, providing valuable insights and capabilities beyond Gymnasium.”