Bias and variance in value function estimation

By Shie Mannor, Duncan Simester, Peng Sun, John N. Tsitsiklis. 2004

88 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Marketing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Marketing

Citations in the corpus

Hyperparameter Selection for Offline Reinforcement Learning (2020, methodology): “These ﬁrst estimate a transition model from the oﬄine data, and then perform policy evaluation in the MDP deﬁned by the transition model Mannor et al. (2004).”
Generalized TD Learning (2011, methodology): “This framework generalizes previously developed model-free algorithms, which include temporal difference learning and its extensions, and moreover, enables us to investigate the statistical properties of these algorithms, which have not been yet elucidated.”
Efficient exploration with Double Uncertain Value Networks (2017, background): “Mannor et al. (2004, 2007) studied two environment sources of variance that may influence the return distribution: the ‘internal variance’ due to a stochastic environment, and the ‘parametric (model) variance’ due to bias in the environment model.”
Distributionally Robust Markov Decision Processes (2010, background): “In practice, parameter uncertainty – the deviation of the model parameters from the true ones (reward r and transition probability p) – often causes the performance of " optimal " policies to degrade significantly [4].”
MDPs with Non-Deterministic Policies (2008, methodology): “We can use some variance estimation on the value function [8] along with a Z-Test to get some confidence level on our comparisons and find the probability of having the wrong order when comparing actions according to their values.”
Percentile Optimization for Markov Decision Processes with Parameter Uncertainty (2010, methodology): “P . Following similar lines as in Mannor et al. (2006) , we focus on finding a stationary policy that performs well according to the second order approximation of the expected return.”
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control (2005, methodology): “We quantify uncertainty using approximate Bayesian versions of each reinforcement learning algorith”
Reinforcement learning with Gaussian processes (2005, background): “For a more thorough discussion of intrinsic and extrinsic uncertainties see Mannor et al. (2004) where N (x, x ) def = ∆V (x) − γ∆V (x ).”
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, methodology): “If your confidence in the value estimate of state s is high, then λ(s) should be close to 0, meaning we trust the estimates provided by v̂.”
Minimax-Optimal Off-Policy Evaluation with Linear Function Approximation (2020, methodology): “This has been studied in the tabular case with bias and variance analysis (Mannor et al., 2004).”