Reverb: A Framework For Experience Replay
By Albin Cassirer, Gabriel Barth-Maron, Eugene Brevdo, Sabela Ramos, Toby Boyd, Thibault Sottiaux, M. Kroiss. 2021
46 citations recorded by the source corpus.
A central component of training in Reinforcement Learning (RL) is Experience: the data used for training. The mechanisms used to generate and consume this data have an important effect on the performance of RL algorithms. In this paper, we introduce Reverb: an efficient, extensible, and easy to use system designed specifically for experience replay in RL. Reverb is designed to work efficiently in distributed configurations with up to thousands of concurrent clients. The flexible API provides users with the tools to easily and accurately configure the replay buffer. It includes strategies for selecting and removing elements from the buffer, as well as options for controlling the ratio between sampled and inserted elements. This paper presents the core design of Reverb, gives examples of how it can be applied, and provides empirical results of Reverb's performance characteristics.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Sergey Levine
- Ziyun Wang
- T. Paine
- Yoshua Bengio
- Matthew W. Hoffman
- Bobak Shahriari
- Caglar Gulcehre
- Pieter Abbeel
Citations in the corpus
- Acme: A Research Framework for Distributed Reinforcement Learning (2020, methodology): “Reverb can be roughly described as a storage system which enables efficient insertion and routing of items and a flexible sampling mechanism that allows: first-in-first-out, last-in-first-out, uniform, and weighted sampling schemes.”
- Launchpad: A Programming Model for Distributed Machine Learning Research (2021, background): “However in situations where this is not the case Launchpad also provides a ReverbNode which can access a dataset provided by Reverb (Cassirer et al., 2021).”