An Analysis of Categorical Distributional Reinforcement Learning

By Mark Rowland, Marc G. Bellemare, Will Dabney, R´emi Munos, Yee Whye Teh. 2018

127 citations recorded by the source corpus.

Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performance. This was demonstrated by the recently proposed C51 algorithm, based on categorical distributional reinforcement learning (CDRL) [Bellemare et al., 2017]. However, the theoretical properties of CDRL algorithms are not yet well understood. In this paper, we introduce a framework to analyse CDRL algorithms, establish the importance of the projected distributional Bellman operator in distributional RL, draw fundamental connections between CDRL and the Cram\'er distance, and give a proof of convergence for sample-based categorical distributional reinforcement learning algorithms.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Implicit Quantile Networks for Distributional Reinforcement Learning (2018, methodology): “Cat-egorical DQN (Bellemare et al., 2017, C51) combines a categorical distribution and the cross-entropy loss with the Cram´er-minimizing projection (Rowland et al., 2018).”
An Introduction to Deep Reinforcement Learning (2018, unknown): “It has been shown that such a distributional Bellman equation can be used in practice, with deep learning as the function approximator (Bellemare et al., 2017; Dabney et al., 2017; Rowland et al., 2018).”
Distributional Deep Reinforcement Learning with a Mixture of Gaussians (2019, background): “the projection operator is proved in [19] in terms of the Cramér distance.”
Fully Parameterized Quantile Function for Distributional Reinforcement Learning (2019, background): “…value-based reinforcement learning in that, instead of focusing only on the expectation of the return, distributional reinforcement learning also takes the intrinsic randomness of returns within the framework into consideration [Bellemare et al., 2017, Dabney et al., 2018b,a, Rowland et al., 2018].”
Distributional reinforcement learning with linear function approximation (2019, methodology): “Rowland et al. (2018) showed that the heuristic projection minimizes a probability metric called the Cram´er distance.”
A Comparative Analysis of Expected and Distributional Reinforcement Learning (2019, methodology): “Later work showed the convergence of stochastic updates in the distributional setting (Rowland et al. 2018).”
Distributional Reinforcement Learning for Efficient Exploration (2019, background): “In the limit, distributional RL captures the intrinsic uncertainty of an MDP (Bellemare et al., 2017; Dabney et al., 2017; 2018; Rowland et al., 2018).”
Learning to Score Behaviors for Guided Policy Optimization (2019, background): “These approaches have impressive experimental results [2, 8], with a growing body of theory [28, 25, 3, 29].”