Distributional Reinforcement Learning with Quantile Regression

By Will Dabney, Mark Rowland, Marc G. Bellemare, R´emi Munos. 2017

1,011 citations recorded by the source corpus.

In reinforcement learning (RL), an agent interacts with the environment by taking actions and observing the next state and reward. When sampled probabilistically, these state transitions, rewards, and actions can all induce randomness in the observed long-term return. Traditionally, reinforcement learning algorithms average over this randomness to estimate the value function. In this paper, we build on recent work advocating a distributional approach to reinforcement learning in which the distribution over returns is modeled explicitly instead of only estimating the mean. That is, we examine methods of learning the value distribution instead of the value function. We give results that close a number of gaps between the theoretical and algorithmic results given by Bellemare, Dabney, and Munos (2017). First, we extend existing results to the approximate distribution setting. Second, we present a novel distributional reinforcement learning algorithm consistent with our theoretical formulation. Finally, we evaluate this new algorithm on the Atari 2600 games, observing that it significantly outperforms many of the recent improvements on DQN, including the related distributional algorithm C51.

Subfields: Artificial Intelligence, Economics and Econometrics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Economics and Econometrics

In Reinforcement Learning

Citations in the corpus

Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “The implicit quantile network (IQN) model (Dabney et al., 2018a) reparameter-izes a sample τ ∼ U ([0 , 1]) through a deterministic function to produce samples from the underlying data distribution.”
Reward learning from human preferences and demonstrations in Atari (2018, background): “Deep RL is very sensitive to the scale of the reward, and this should be alleviated with quantile distributional RL while improving performance (Dabney et al., 2017).”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “Quantile regression has been proposed as an alternative solution to the minimization of the Wasserstein (Dabney et al., 2018b), (Dabney et al., 2018a), and has shown to perform well for Distributional RL.”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “When only using only 1% or 10% of the data, in setting (2) (Table 3), CQL substantially outperforms REM and QR-DQN, especially in the harder 1% condition, achieving 36x and 6x times the return of the best prior method on Q*bert and Breakout, respectively.”
Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “Expectile regression is closely related to quantile regression (Koenker & Hallock, 2001), which is a popular technique for estimating quantiles of a distribution widely used in reinforcement learning (Dabney et al., 2018b;a) .”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “Our exploration policy is a form of distributional RL, (Bellemare et al., 2017; Dabney et al., 2017, 2018), though we found that the Huber quantile regression loss used in existing distributional RL work scaled poorly with dimension and was unable to learn the sparse multimodal 4-dimensional…”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “Distributional RL [4, 13, 12] proposes using a distribution to represent optimal action-values rather than a point estimate.”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “To reveal the effects behind these modiﬁcations, we QB-SAC is B-SAC with Quantile distributional networks (Dabney et al., 2018b).”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, methodology): “The human-normalized scores reported in this paper are given by the formula (van Hasselt et al., 2016; Dabney et al., 2018) score = agent − random human − random , where agent , human and random are the per-game raw scores (undiscounted returns) for the given agent, a reference human player, and…”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, methodology): “Finally, distributional methods (Bellemare et al., 2017; Dabney et al., 2018b) replace the scalar prediction made by DQN with a value distribution, and accordingly introduce new losses to the algorithm.”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “Similar to QR-DQN, the output of the Q-network is a K -dimensional vector.”
Discovering Reinforcement Learning Algorithms (2020, positive): “inspired by the success of Distributional RL [2, 7].”
Mastering Atari with Discrete World Models (2020, background): “Several works have extended DQN to incorporate bias correction as in DDQN (Van Hasselt et al., 2015), prioritized experience replay (Schaul et al., 2015), architectural improvements (Wang et al., 2016), and distributional value learning (Bellemare et al., 2017; Dabney et al., 2017; 2018).”
Revisiting Fundamentals of Experience Replay (2020, background): “…& Stone, 2015; Kapturowski et al., 2019), learning algorithms using auxiliary tasks (Jaderberg et al., 2016; Fedus et al., 2019), distributional variants of RL (Bellemare et al., 2017; Dabney et al., 2018), and the use of prioritisation in sampling from experience replay (Schaul et al., 2015).”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, methodology): “Particularly, Dabney et al. (2017) used a set of quantiles to approximate this value distribution.”
Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping (2019, methodology): “In the following sections, we present two versions of Q2Opt based on two recently introduced distributional algorithms: QR-DQN and IQN.”