A Comparative Analysis of Expected and Distributional Reinforcement Learning

By Clare Lyle, Pablo Samuel Castro, Marc G. Bellemare. 2019

97 citations recorded by the source corpus.

Since their introduction a year ago, distributional approaches to reinforcement learning (distributional RL) have produced strong results relative to the standard approach which models expected values (expected RL). However, aside from convergence guarantees, there have been few theoretical results investigating the reasons behind the improvements distributional RL provides. In this paper we begin the investigation into this fundamental question by analyzing the differences in the tabular, linear approximation, and non-linear approximation settings. We prove that in many realizations of the tabular and linear approximation settings, distributional RL behaves exactly the same as expected RL. In cases where the two methods behave differently, distributional RL can in fact hurt performance when it does not induce identical behaviour. We then continue with an empirical analysis comparing distributional and expected RL methods in control settings with non-linear approximators to tease apart where the improvements from distributional RL methods are coming from.

Subfields: Artificial Intelligence, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Distributional RL (Bellemare et al., 2017a), which was not initially introduced as a representation learning technique, has been shown by Lyle et al. (2019) to only play an auxiliary task role.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Despite these algorithmic advances, our understanding of performance gains due to practical distributional RL methods remains limited [38, 6].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “28 However, we found very few papers with this characteristic[17,170,208] — we note that this is not encouraged in the ML community; moreover, negative results reduce the chance of paper acceptance [197].”
Distributional reinforcement learning in the brain (2020, background): “, single-layer neural networks) do not learn hidden representations, so distributional RL confers no benefit for estimating the expected value in the linear setting [68].”
Munchausen Reinforcement Learning (2020, background): “It appears that the benefits of distRL do not really come from RL principles, but rather from the regularizing effect of modelling a distribution and its role as an auxiliary task in a deep learning context [21].”
Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors (2020, background): “Besides, compared with most distributional RL algorithms that can only handle discrete and lowdimensional action spaces [18]–[22], DSAC is applicable to continuous control settings by optimizing an independent stochastic policy network.”
Policy Evaluation Networks (2020, methodology): “Predicting buckets instead of exact real values provides a regularization effect, similar to the idea of learning with auxiliary tasks (Caruana, 1997; Sutton et al., 2011; Jaderberg et al., 2017c; Lyle et al., 2019).”