Parametric Return Density Estimation for Reinforcement Learning

By Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki TANAKA. 2010

120 citations recorded by the source corpus.

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected returns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

Citations in the corpus

Distributional Reinforcement Learning with Quantile Regression (2017, neutral): “Recently, Bellemare, Dabney, and Munos (2017) showed that the distribution of the random returns, whose expectation constitutes the aforementioned value, can be described by the distributional analogue of Bellman’s equation, echoing previous results in risk-sensitive reinforcement learning (Heger 19”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “Distributional reinforcement learning (Jaquette, 1973; Sobel, 1982; White, 1988; Morimura et al., 2010b; Bellemare et al., 2017) focuses on the intrinsic randomness of returns within the reinforcement learning (RL) framework.”
A Distributional Perspective on Reinforcement Learning (2017, methodology): “Although the Gaussian case has previously been considered (Morimura et al., 2010a; Tamar et al., 2016), to the best of our knowledge we are the ﬁrst to use a rich class of parametric distributions.”
Distributed Distributional Deterministic Policy Gradients (2018, background): “Although the idea of estimating a distribution over returns has been revisited before (Sobel, 1982; Morimura et al., 2010), Bellemare et al. demonstrated that this estimation alone was enough to achieve state-of-the-art results on the Atari 2600 benchmarks.”
Batch Mode Reinforcement Learning based on the Synthesis of Artificial Trajectories (2012, background): “Notice that this type of criterion has received more and more attention during the last few years inside the RL community (Defourny et al. 2008; Morimura et al. 2010a, 2010b).”
Distributional Reinforcement Learning for Efficient Exploration (2019, background): “For instance Morimura et al. (2012) showed in a simple environment of Cliff Walk the policy maximizing low quantiles yields paths further away from the dangerous cliff.”
Efficient exploration with Double Uncertain Value Networks (2017, background): “Morimura et al. (2012) studied parametric return distribution propagation as well.”
Distributional Deep Reinforcement Learning with a Mixture of Gaussians (2019, background): “Studies on the distributional perspective before [3] had subordinated its purpose on modeling parametric uncertainty [11], designing risk-sensitive algorithms [12], [13], or for theoretical analysis [14].”
A comprehensive survey on safe reinforcement learning (2015, methodology): “Finally, other approaches are based on the use of optimization criteria falling into the area of financial engineering, such as the r-squared, value-at-risk (VaR) (Mausser and Rosen, 1998; Kashima, 2007; Luenberger, 2013), or the density of the return (Morimura et al., 2010a,b) (Section 3.4).”
Learning the Variance of the Reward-To-Go (2016, background): “More recently, Morimura et al. (2010) proposed TD learning rules for a parametric distribution of the return, albeit without function approximation nor formal guarantees.”
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, background): “In practice, however, this often leads to high variance, and thus we mitigate the variance by choosing λ less than one and speed learning overall, but introduce bias.”
An Analysis of Categorical Distributional Reinforcement Learning (2018, background): “The moments or distribution of the random return have also been considered in the literature, with a variety of approaches proposing algorithms for estimating more complex distributional information [Morimura et al., 2010b,a, Prashanth and Ghavamzadeh, 2013 et al., 2016].”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, methodology): “Particularly, Morimura et al. (2012) used a small quantile for control, resulting in a safe policy in the cliff grid world.”