Nonparametric Return Distribution Approximation for Reinforcement Learning

By Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki TANAKA. 2010

125 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “Distributional reinforcement learning (Jaquette, 1973; Sobel, 1982; White, 1988; Morimura et al., 2010b; Bellemare et al., 2017) focuses on the intrinsic randomness of returns within the reinforcement learning (RL) framework.”
Decentralized Likelihood Quantile Networks for Improving Performance in Deep Multi-Agent Reinforcement Learning (2018, background): “Distributional RL has also been studied for designing risk sensitive algorithms (Morimura et al., 2010).”
Benchmarking Batch Deep Reinforcement Learning Algorithms (2019, methodology): “Quantile Regression DQN (QR-DQN) [Dabney et al., 2017] is a distributional reinforcement learning method [Morimura et al., 2010, Bellemare et al., 2017] which aims to estimate the set of K τ -quantiles of the return distribution, { τ } K = { i +0 .”
Parametric Return Density Estimation for Reinforcement Learning (2010, methodology): “Thus, Pr(E≤η | s, a) = ∑ s+1∈S pT(s+1|s, a) ∫ r pR(r|s, a, s+1) × ∑ a+1∈A π(a+1|s+1) Pr ( E+1≤ η − r γ | s+1, a+1 ) dr, 1The distributional Bellman equation in Morimura et al. (2010) is for the cumulative density, not for the density. holds.”
Distributional Deep Reinforcement Learning with a Mixture of Gaussians (2019, background): “Studies on the distributional perspective before [3] had subordinated its purpose on modeling parametric uncertainty [11], designing risk-sensitive algorithms [12], [13], or for theoretical analysis [14].”
Risk-Constrained Reinforcement Learning with Percentile Risk Criteria (2015, background): “…the works in Tamar et al. (2012); Prashanth and Ghavamzadeh (2013) present RL algorithms for several variance-related risk measures, the works in Morimura et al. (2010); Tamar et al. (2015); Petrik and Subramanian (2012) consider CVaR-based formulations, and the works in Tallec (2007); Shapiro…”
Fully Parameterized Quantile Function for Distributional Reinforcement Learning (2019, background): “Distributional reinforcement learning [Jaquette et al., 1973, Sobel, 1982, White, 1988, Morimura et al., 2010, Bellemare et al., 2017] differs from value-based reinforcement learning in that, instead of focusing only on the expectation of the return, distributional reinforcement learning also takes…”
An Introduction to Deep Reinforcement Learning (2018, unknown): “This approach has the following advantages: • It is possible to implement risk-aware behavior (see e.g., Morimura et al., 2010).”
Algorithms for CVaR Optimization in MDPs (2014, methodology): “[19] estimate the return distribution while exploring using a CVaR-based risksensitive policy.”
An Analysis of Categorical Distributional Reinforcement Learning (2018, background): “The moments or distribution of the random return have also been considered in the literature, with a variety of approaches proposing algorithms for estimating more complex distributional information [Morimura et al., 2010b,a, Prashanth and Ghavamzadeh, 2013 et al., 2016].”
Optimizing the CVaR via Sampling (2014, background): “Morimura et al. (2010) consider the expected return, with a CVaR based risk-sensitive policy for guiding the exploration while learning.”
A comprehensive survey on safe reinforcement learning (2015, methodology): “Finally, other approaches are based on the use of optimization criteria falling into the area of financial engineering, such as the r-squared, value-at-risk (VaR) (Mausser and Rosen, 1998; Kashima, 2007; Luenberger, 2013), or the density of the return (Morimura et al., 2010a,b) (Section 3.4).”
Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning (2018, background): “Distributional RL has also been studied for designing risk sensitive algorithms [23].”