The variance of discounted Markov decision processes

By M. J. Sobel. 1982

Subfields: Reinforcement Learning, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Game & Decision Theory

Citations in the corpus

RUDDER: Return Decomposition for Delayed Rewards (2018, background): “P a0ˇ(a 0js0). The local variance is Var s0;a0[q ˇ(s0;a0) js;a] = X s0 p(s0js;a) X a0 ˇ(a0js0) qˇ(s0;a0) 2 (A186) X s0 p(s0js;a) X a0 ˇ(a0js0) qˇ(s0;a0) !2 : This result is Equation (6) in Sobel 1982 [121]. Sobel derived these formulas also for ﬁnite horizons and an analog formula if the reward depe”
The Uncertainty Bellman Equation and Exploration (2017, neutral): “The observation that the higher moments of a value function also satisfy a form of Bellman equation is not new and has been observed by some of the early papers on the subject (Sobel, 1982).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, background): “inty, Bayesian uncertainty, and model uncertainty. Epistemic uncertainty is distinct from the so-called aleatoric uncertainty or risk, which is the natural randomness of the process we are observing (Sobel, 1982; Tamar et al., 2016; Bellemare et al., 2017). This difference can be made clear by an ex”
Implicit Quantile Networks for Distributional Reinforcement Learning (2018, background): “Distributional reinforcement learning (Jaquette, 1973; Sobel, 1982; White, 1988; Morimura et al., 2010b; Bellemare et al., 2017) focuses on the intrinsic randomness of returns within the reinforcement learning (RL) framework.”
A Distributional Perspective on Reinforcement Learning (2017, methodology): “Combining with (10) we obtain Proposition 1 (Sobel, 1982) .”
Distributed Distributional Deterministic Policy Gradients (2018, background): “Although the idea of estimating a distribution over returns has been revisited before (Sobel, 1982; Morimura et al., 2010), Bellemare et al. demonstrated that this estimation alone was enough to achieve state-of-the-art results on the Atari 2600 benchmarks.”
PAC Bounds for Discounted MDPs (2012, methodology): “Lemma 8 (Sobel 1982).”
On the Sample Complexity of Reinforcement Learning with a Generative Model (2012, background): “…the sum of discounted rewards, like the expected value of the sum (value function), satisﬁes a Bellman-like equation, in which the variance of the value function plays the role of the instant reward (Munos & Moore, 1999; Sobel, 1982), to derive a sharp bound on the variance of the value function.”
Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (2019, background): “This novel decomposition can be thought of as a generalization of the celebrated Bellman-equation of variance [Sobel, 1982] in the off-policy, episodic MDP setting with a ﬁnite sample and can be of independent interest.”
Efficient exploration with Double Uncertain Value Networks (2017, background): “3) is certainly not new Sobel (1982); White (1988), nearly all RL research has focussed on the mean action-value.”
Biases and Variance in Value Function Estimates (2004, background): “In particular, Sobel (1982) provides an expression for the internal variance in a Markov Decision Process with discounted rewards, while Filar et al. (1989) and Baukal-Gursoy and Ross (1992) consider the average reward criterion.”
Mean-Variance Optimization in Markov Decision Processes (2011, background): “We note that expressions for the variance of the discounted reward for stationary policies were developed in Sobel (1982).”
Policy Gradients with Variance Related Risk Criteria (2012, methodology): “We remark that similar equations for the infinite horizon discounted return case were presented by Sobel (1982), in which I−P ′ is replaced with I−βP , where β 1 is the discount factor.”
The optimal unbiased value estimator and its relation to LSTD, TD and MC (2009, methodology): “C estimator with n paths available for estimation: V [V¯(n)] = MSE[V¯(n)] = 1 n [R] and P (|V¯(n)− V | ≥ ǫ) ≤ 2exp „ − ǫ2n 2 V [R]+ 2dǫ/3 « , where V [R] is the variance in the cumulated reward (see (Sobel, 1982) for the variance of a MRP) and d is an upper bound for the cumulated reward of any path”
Actor-Critic Algorithms for Risk-Sensitive MDPs (2013, methodology): “From the Bellman equation of Λ(x), proposed by Sobel [19], it is straightforward to derive Bellman equations for U(x) and the square reward action-value function W(x, a) 4 = E [ D(x, a)(2) ] (see Appendix B.”
Bias and variance in value function estimation (2004, unknown): “An expression for the variance of Z for discounted reward was given by Sobel (1982).”
Risk-Averse Bayes-Adaptive Reinforcement Learning (2021, background): “Another line of work considers the mean-variance criterion (Sobel, 1982), but this metric has undesirable properties as it is not a coherent risk metric.”
Fully Parameterized Quantile Function for Distributional Reinforcement Learning (2019, background): “Distributional reinforcement learning [Jaquette et al., 1973, Sobel, 1982, White, 1988, Morimura et al., 2010, Bellemare et al., 2017] differs from value-based reinforcement learning in that, instead of focusing only on the expectation of the return, distributional reinforcement learning also takes…”