A Distributional Perspective on Reinforcement Learning

By Marc G. Bellemare, Will Dabney, R´emi Munos. 2017

1,894 citations recorded by the source corpus.

In this paper we argue for the fundamental importance of the value distribution: the distribution of the random return received by a reinforcement learning agent. This is in contrast to the common approach to reinforcement learning which models the expectation of this return, or value. Although there is an established body of literature studying the value distribution, thus far it has always been used for a specific purpose such as implementing risk-aware behaviour. We begin with theoretical results in both the policy evaluation and control settings, exposing a significant distributional instability in the latter. We then use the distributional perspective to design a new algorithm which applies Bellman's equation to the learning of approximate value distributions. We evaluate our algorithm using the suite of games from the Arcade Learning Environment. We obtain both state-of-the-art results and anecdotal evidence demonstrating the importance of the value distribution in approximate reinforcement learning. Finally, we combine theoretical and empirical evidence to highlight the ways in which the value distribution impacts learning in the approximate setting.

Subfields: Artificial Intelligence, Management Information Systems, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Information Systems

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reward learning from human preferences and demonstrations in Atari (2018, methodology): “In addition to the experiments presented here, we were unsuccessful at achieving significant performance improvements from a variety of other ideas: distributional RL (Bellemare et al., 2017), quantile distributional RL (Dabney et al.”
Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “, 2018), and distributional methods (Bellemare et al., 2017; Barth-Maron et al., 2018).”
When to use parametric models in reinforcement learning? (2019, background): “…with double Q-learning [van Hasselt, 2010, van Hasselt et al., 2016], dueling network architectures [Wang et al., 2016], prioritised experience replay [Schaul et al., 2016], noisy networks for exploration [Fortunato et al., 2017], and distributional reinforcement learning [Bellemare et al., 2017].”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “For additional hyperparameter details, see Table 1 and (Bellemare et al., 2017a).”
Distributed Prioritized Experience Replay (2018, methodology): “In Figure 2, on the left, we compare the median human normalized score across all 57 games to several baselines: DQN, Prioritized DQN, Distributional DQN (Bellemare et al., 2017), Rainbow, and Gorila.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “Recently Bellemare, Dabney, and Munos (2017) proposed to model such distributions with probability masses placed on a discrete support z , where z is a vector with N atoms ∈ N + atoms , deﬁned by N atoms − 1 for i ∈ { 1 , . . . , N atoms } .”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “…to use auxiliary tasks that allow agents to start learning a representation before the first extrinsic reward is observed (Jaderberg et al., 2017); the distributions output by the C51 algorithm of Bellemare, Dabney, and Munos (2017) may be viewed as a particularly meaningful set of auxiliary tasks.”
Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning (2020, neutral): “Lu and Amato (2019) utilise Distributional RL (Bellemare et al., 2017) in combination with Hysteretic Q -learning in order to also distinguish between miscoordination and stochasticity, and argue their approach is more stable and robust to hyperparameters than the above methods.”
Wireless Network Intelligence at the Edge (2018, background): “Such distributional RL outperforms stateof-the-art RL methods including deep Q-learning and asynchronous actor-critic RL [144], [145].”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “ble DQN (DDQN) [121, 122] which helps exploration as the overestimation bias is reduced. Noisy DQNs [20] explore by a stochastic layer in the policy network (see [40, 101]). Distributional Q-learning [7] proﬁts from noise since means that have high variance are more likely selected. The dueling netw”
Meta-Gradient Reinforcement Learning (2018, background): “, 2016b], multi-step targets [Sutton, 1988, Sutton and Barto, 2018], distributional RL [Bellemare et al., 2017], and parameter noise for exploration [Fortunato et al.”
COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “As mentioned in Section 6, we initially tried using a multi-dimensional version of the Huber quantile function loss typically used in distributional RL, but this was very slow to train.”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, background): “efficient, stable, and high-performing reinforcement learning algorithms [32].”
A Framework for Data-Driven Robotics (2019, methodology): “As in D4PG, instead of using a scalar Q function, we adopt a distributional value function [70].”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “Distributional RL [4, 13, 12] proposes using a distribution to represent optimal action-values rather than a point estimate.”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “Our baseline is the C51 distributional reinforcement learning agent (Bellemare, Dabney, and Munos 2017), and we use published hyperparameters unless otherwise noted.”
The Uncertainty Bellman Equation and Exploration (2017, neutral): “, the uncertainty due to estimating a parameter using a finite amount of data, rather than the higher moments of the reward-to-go (Lattimore & Hutter, 2012; Azar et al., 2012; Mannor & Tsitsiklis, 2011; Bellemare et al., 2017).”
Exploration by Random Network Distillation (2018, background): “Bellemare et al. (2017) parametrize the value as a linear combination of value heads that estimate probabilities of discretized returns.”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “odel uncertainty. Epistemic uncertainty is distinct from the so-called aleatoric uncertainty or risk, which is the natural randomness of the process we are observing (Sobel, 1982; Tamar et al., 2016; Bellemare et al., 2017). This difference can be made clear by an example. Consider ﬂipping a biased ”
Distributional Reinforcement Learning with Quantile Regression (2017, methodology): “ L2 distance between Qand TQ. Unfortunately, another result shows that we cannot in general minimize the Wasserstein metric (viewed as a loss) using stochastic gradient descent. Theorem 1 (Theorem 1, Bellemare et al. 2017). Let Y^m:= 1 m Pm i=1 Yi be the empirical distribution derived from samples Y”