Dopamine: A Research Framework for Deep Reinforcement Learning

By Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, Marc G. Bellemare. 2018

296 citations recorded by the source corpus.

Deep reinforcement learning (deep RL) research has grown significantly in recent years. A number of software offerings now exist that provide stable, comprehensive implementations for benchmarking. At the same time, recent deep RL research has become more diverse in its goals. In this paper we introduce Dopamine, a new research framework for deep RL that aims to support some of that diversity. Dopamine is open-source, TensorFlow-based, and provides compact and reliable implementations of some state-of-the-art deep RL agents. We complement this offering with a taxonomy of the different research objectives in deep RL research. While by no means exhaustive, our analysis highlights the heterogeneity of research in the field, and the value of frameworks such as ours.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Our implementations of the proposed algorithms are based on Dopamine (Castro et al., 2018).”
Model-Based Reinforcement Learning for Atari (2019, methodology): “It can be seen that our method is more sample-efﬁcient than a highly tuned Rainbow baseline on almost all games, requires less than half of the samples on more than half of the games and, on Freeway , is more than 10x more sample-efﬁcient.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “Dopamine [38]: Offical DQN implementation https://github.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “As mentioned in Dopamine’s [11] GitHub repository, changing these parameters can signiﬁcantly affect performance, without necessarily being indicative of an algorithmic difference.”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, methodology): “For our learning agents, we extend the implementations found in Dopamine [14].”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, background): “, 2020), Reverb (Albin Cassirer, Gabriel Barth-Maron, Thibault Sottiaux, Manuel Kroiss, Eugene Brevdo, 2020), Dopamine (Castro et al., 2018) and TensorFlow (Abadi et al.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Dopamine (Castro et al., 2018) is a framework focusing on single-process agents in the DQN (Mnih et al.”
The Hanabi Challenge: A New Frontier for AI Research (2019, methodology): “ears, resulting in a learning algorithm that is both sample ecient and achieves high rewards at convergence. In our benchmark we use a multi-agent version of Rainbow, based on the Dopamine framework [38]. In our code the agents controlling the dierent players share parameters. Our Rainbow agent is f”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, methodology): “Other published research code bases include OpenAI Baselines [28] and Dopamine [29], both of which are implemented in Tensorﬂow [30], and neither of which are optimized to the extent of rlpyt nor contain all three algorithm families.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “We use standard preprocessing to image frames and environment rewards [Mnih et al., 2015, Castro et al., 2018], and match the recommendations of Machado et al. [2018] for fair and reproducible results.”
Avalanche: an End-to-End Library for Continual Learning (2021, background): “Other relevant projects in the ﬁeld of reinforcement learning are Dopamine [5], which focuses on simplicity and easy prototyping, and project Malmo [24], which is based on the famous Minecraft game.”
Dream to Control: Learning Behaviors by Latent Imagination (2019, methodology): “For Atari, we show the ﬁnal performance of SimPLe (Kaiser et al., 2019), DQN (Mnih et al., 2015) and Rainbow (Hessel et al., 2018) reported by Castro et al. (2018), and for DeepMind Lab that of IMPALA (Espeholt et al., 2018) as a guideline.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “AI Gym [299] have allowed single-agent RL to move beyond toy domains. For DRL there are open-source frameworks that provide compact and reliable implementations of some state-of-theart DRL algorithms [300]. Even though MDRL is a recent area, there are now a number of open sourced simulators and benc”
RecSim: A Configurable Simulation Platform for Recommender Systems (2019, background): “To be self-contained, R EC S IM also provides an adapter for applying the DQN agent in Dopamine to the simulation environments packaged with R EC S IM .”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, methodology): “Rather than training a full Q-network using C ON QUR, we leverage pre-trained networks from the Dopamine package (Castro et al., 2018), 7 and use C ON QUR to learn ﬁnal layer weights, i.e., a new “linear approximator” w.r.t. the learned feature representation.”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “For ALE, we used the individual runs from Dopamine [14] baselines except for DreamerV2 [38], REM [1] and M-IQN [109], for which the individual run scores were obtained from the corresponding authors.”
Ranking Policy Gradient (2019, methodology): “Implementations of all methods are provided in the Dopamine framework (Castro et al., 2018).”