Bias-Variance Error Bounds for Temporal Difference Updates
By Michael Kearns, Satinder Singh. 2000
108 citations recorded by the source corpus.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Meta-Gradient Reinforcement Learning (2018, methodology): “Kearns and Singh [2000] derive upper bounds on the error of temporal-difference algorithms, and use these bounds to derive schedules for λ .”
- Boosting the Actor with Dual Critic (2017, methodology): “In this subsection, we will extend the minimax game between the actor and critic to the multi-step setting, which has been widely utilized in temporal-difference algorithms for better bias/variance tradeoffs (Sutton and Barto, 1998; Kearns and Singh, 2000).”
- Off-policy learning with eligibility traces: a survey (2013, background): “This leads to some known algorithms – offpolicy LSTD(λ), LSPE(λ), TD(λ), TDC/GQ(λ) – and suggests new extensions – offpolicy FPKF(λ), BRM(λ), gBRM(λ), GTD2(λ).”
- Q(λ) with Off-Policy Corrections (2016, background): “It is well-known that λ trades off the bias from bootstrapping with an approximate Q-function, with the variance from using a sampled multi-step return [4], with intermediate values of λ usually performing best in practice [15, 13].”
- Inequity aversion improves cooperation in intertemporal social dilemmas (2018, background): “Learning the association between short-term actions and long-term consequences is a high-variance and error-prone process, both for animals [37] and reinforcement learning algorithms [38].”
- Q( λ ) with Off-Policy Corrections (background): “It is well-known that λ trades off the bias from bootstrapping with an approximate Q-function, with the variance from using a sampled multi-step return [4], with intermediate values of λ usually performing best in practice [15, 13].”
- Stochastic Variance Reduction Methods for Policy Evaluation (2017, unknown): “(Singh & Sutton, 1996; Kearns & Singh, 2000).”
- The optimal unbiased value estimator and its relation to LSTD, TD and MC (2009, background): “Kearns and Singh (2000) present upper bounds on the generalization error for Temporal Difference estimators (TD).”
- A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, background): “…produced by an adaptive learning agent’s interaction with its environment, which makes tuning the parameters of the learning Appears in: Proceedings of the 15th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2016), J. Thangarajah, K. Tuyls, C. Jonker, S. Marsella (eds.)”
- Safe and Efficient Off-Policy Reinforcement Learning (2016, background): “Intermediate values of λ trade off estimation bias with sample variance (Kearns and Singh, 2000).”