Bias and Variance Approximation in Value Function Estimates

By Shie Mannor, Duncan Simester, Peng Sun, John N. Tsitsiklis. 2007

211 citations recorded by the source corpus.

Subfields: Management Information Systems, Management Science and Operations Research, Marketing, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Management Information Systems

In Management Science and Operations Research

In Marketing

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, neutral): “Function approximation error and its effect on bias and variance in reinforcement learning algorithms have been studied in prior works (Pendrith et al., 1997; Mannor et al., 2007).”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “earning. Grünewälder & Obermayer [41] investigated the bias of temporal difference learning (TD), Monte Carlo estimators (MC), and least-squares temporal difference learning (LSTD). Mannor et al. [77] and O’Donoghue et al. [88] derived bias and variance expressions for updating Q-values. The tru”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, background): “Modelbased estimates are low variance (Mannor et al., 2007) but are biased and not consistent estimators when the model representation chosen does not match the true environment.”
Doubly Robust Off-policy Value Evaluation for Reinforcement Learning (2015, background): “When an exact state representation is used and each state-action pair appears sufﬁciently often in the data, such regres-1 sion estimators have provably low variances and negligible biases (Mannor et al., 2007), and often outperform alternatives in practice (Paduraru, 2013).”
The Uncertainty Bellman Equation and Exploration (2017, methodology): “ of the uncertainty Bellman equation (UBE), where we deﬁne uncertainty to be the variance of the value estimator the agent is learning, in a sense similar to the parametric variance of Mannor et. al. [17]. Intuitively speaking, if the agent has high uncertainty (as measured by high estimator varianc”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, background): “Another line of research in RL, related to confidence intervals, is the estimation of variance , for example estimating the variance of the value function of a particular policy (Mannor et al. 2007; Tetreault et al. 2007).”
Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes (2019, methodology): “For example, one can use model-based estimates (Mannor et al., 2007) or estimate the q-function directly using fitted LSTDQ or more general q-iteration (Antos et al.”
Off-Policy Evaluation via Off-Policy Classification (2019, methodology): “In MDP modeling, a model is ﬁtted to D , and ⇡ is rolled out in the learned model to estimate average return [13, 24].”
Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting (2019, methodology): “In practice, an unbiased estimate of the LHS can be obtained via Monte Carlo averaging of log-importance weights based on D train .”
Safe Reinforcement Learning (2015, methodology): “This model can then be used to estimate the value of any policy (Mannor et al., 2007).”
Efficient exploration with Double Uncertain Value Networks (2017, background): “Mannor et al. (2004, 2007) studied two environment sources of variance that may influence the return distribution: the ‘internal variance’ due to a stochastic environment, and the ‘parametric (model) variance’ due to bias in the environment model.”
Convex Optimization: Algorithms and Complexity (2015, unknown): “[Mannor et al., 2007] Using the same notations as Theorem 6.1, the second moment (under the posterior) of V µ satisfies Epost [ V µV µ> ] = V̂ µV̂ µ > + X̂ ( γ2 ( Q̂V̂ µRµ> +RµV̂ µ > Q̂> ) + γ ( B̂Rµ> +RµB̂> ) + Ŵ ) X̂> + Lvar, where Ŵ is a diagonal matrix with elements Ŵi,i = ∑ a µ(a|i)2 ( γV µ…”
Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach (2015, background): “, fluctuations around the mean), nor its sensitivity to modeling errors, which may significantly affect overall performance [12].”
Bayesian Reinforcement Learning (2010, methodology): “This idea was ﬁrst pursued by Mannor et al (2007), who considered the bias and variance of the value function estimate of a single policy.”
Distributionally Robust Markov Decision Processes (2010, unknown): “[24]).”
Quantifying Uncertainty in Batch Personalized Sequential Decision Making (2014, background): “Mannor et al. (2007) quantify uncertainty of a policy by estimating the bias and variance of the expected value of a policy using estimated discrete transition and reward functions.”
Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Average (2013, background): “Sometimes the bias is mentioned without explanation [Kohavi, 1995], and sometimes it is even claimed that CV is unbiased [Mannor et al., 2007].”
Scaling Up Robust MDPs by Reinforcement Learning (2013, background): “It is thus not surprising that the actual performance of the chosen strategy can signiﬁcantly differ from the model’s prediction due to such parameter uncertainty – the deviation of the model parameters from the true ones (see experiments in [13]).”
Count-Based Exploration with the Successor Representation (2018, background): “ote a diagonal matrix such that N(s;s) = 1 n(s)+1:For a given 0 <1, the following approximation relationship holds: ~ ˇe ˇ e ^ ˇNe 1 The proof of the theorem uses the following lemma. Lemma 3.1 (Mannor et al., 2007). Consider two transition probability matrices P;P0with P0= P+ and successor repr”