Implicit Quantile Networks for Distributional Reinforcement Learning

By Will Dabney, Georg Ostrovski, David Silver, R´emi Munos. 2018

714 citations recorded by the source corpus.

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regression to approximate the full quantile function for the state-action return distribution. By reparameterizing a distribution over the sample space, this yields an implicitly defined return distribution and gives rise to a large class of risk-sensitive policies. We demonstrate improved performance on the 57 Atari 2600 games in the ALE, and use our algorithm's implicitly defined distributions to study the effects of risk-sensitive policies in Atari games.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Quantile regression has been proposed as an alternative solution to the minimization of the Wasserstein (Dabney et al., 2018b), (Dabney et al., 2018a), and has shown to perform well for Distributional RL.”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “Expectile regression is closely related to quantile regression (Koenker & Hallock, 2001), which is a popular technique for estimating quantiles of a distribution widely used in reinforcement learning (Dabney et al., 2018b;a) .”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “Our exploration policy is a form of distributional RL, (Bellemare et al., 2017; Dabney et al., 2017, 2018), though we found that the Huber quantile regression loss used in existing distributional RL work scaled poorly with dimension and was unable to learn the sparse multimodal 4-dimensional…”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “To reveal the effects behind these modiﬁcations, we QB-SAC is B-SAC with Quantile distributional networks (Dabney et al., 2018b).”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “IQN Implicit Quantile Networks (Dabney et al., 2018) is an online distributional RL algorithm that approximates the return distribution using the full quantile function, a continuous map from probabilities to returns.”
Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “, 2018), and Implicit Quantile Networks (Dabney et al., 2018).”
rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch (2019, background): “• Deep Q-Learning : DQN [1] + variants: Double [10], Dueling [11], Categorical [12], Rainbow [13] (minus Noisy Nets), Recurrent (R2D2-like) [6], including vector-valued epsilon-greedy (Ape-X-like) [14] ( coming soon : Implicit Quantile DQN [15]).”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, methodology): “In particular, we extend Implicit Quantile Networks (IQN) (Dabney et al., 2018) to multi-agent settings, as learning state-action distributions is a more robust learning task and captures auxiliary expectations.”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, methodology): “We provide complete implementations of four established agents: DQN (Mnih et al., 2015) (which is the base class for all agents), C51 (Bellemare et al., 2017), a simpliﬁed version of the single-GPU Rainbow agent (Hessel et al., 2018), and IQN (Dabney et al., 2018a).”
Challenges of Real-World Reinforcement Learning (2019, positive): “This approach has been combined with DQN (Mnih et al., 2015) and demonstrated on Atari (Hester et al., 2018), as well as combined with DDPG (Lillicrap et al., 2015) for insertion tasks on robots (Vecer´ık et al.).”
A Comparative Analysis of Expected and Distributional Reinforcement Learning (2019, methodology): “2017) and implicit quantile networks (Dabney et al. 2018), with many of these improving on the results of C51.”
A Theoretical Analysis of Deep Q-Learning (2019, background): “Wang et al., 2016), deep recurrent Q-network (Hausknecht and Stone, 2015), asynchronous DQN (Mnih et al., 2016), andvariants designed for distributionalreinforcement learning (Bellemare et al., 2017; Dabney et al., 2018b,a). All of these algorithms are corroborated only by numerical experiments, wit”
Single-Model Uncertainties for Deep Learning (2018, background): “These considered the estimation of individual quantile levels [83, 78], or unconditional quantile estimates with no applications to uncertainty estimation [16, 59].”
Recurrent Experience Replay in Distributed Reinforcement Learning (2018, background): “Currently, state of the art for a single actor is achieved by the recent distributional reinforcement learning algorithms IQN (Dabney et al., 2018) and Rainbow (Hessel et al., 2018a), and for multi-actor results, Ape-X (Horgan et al., 2018).”
Fully Parameterized Quantile Function for Distributional Reinforcement Learning (2019, methodology): “…value-based reinforcement learning in that, instead of focusing only on the expectation of the return, distributional reinforcement learning also takes the intrinsic randomness of returns within the framework into consideration [Bellemare et al., 2017, Dabney et al., 2018b,a, Rowland et al., 2018].”
Discovering faster matrix multiplication algorithms with reinforcement learning (2022, background): “The value head is an implicit quantile network (IQN) head [15] with 8 samples, and it ignores the policy embedding input z1.”