TD_gamma: Re-evaluating Complex Backups in Temporal Difference Learning

By George Dimitri Konidaris, Scott Niekum, Philip S. Thomas. 2011

24 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Meta-Gradient Reinforcement Learning (2018, background): “Many researchers have sought to automate the selection of these parameters [Kearns and Singh, 2000, Downey and Sanner, 2010, Konidaris et al., 2011, White and White, 2016].”
Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, background): “Recent work has suggested that the λ-return could be replaced by more statistically principled complex returns like the γ-return (Konidaris et al., 2011) or Ωreturn (Thomas et al., 2015).”
Consistent On-Line Off-Policy Evaluation (2017, methodology): “Subsequently, in Konidaris et al. (2011) Assumption 4 was relaxed and instead a closed form approximation of the variance was proposed.”
On the Analysis of Complex Backup Strategies in Monte Carlo Tree Search (2016, methodology): “(6) TD γ (Konidaris et al., 2011) is another complex backup learning algorithm which uses the parameter-free γ-return to update state value estimates, and defines the following weights to compute the complex return in (5): w γ n,L = ( n i=1 γ 2(i−1) ) −1 L n=1 ( n i=1 γ 2(i−1) ) −1 , (7) where γ is…”
True Online Temporal-Difference Learning (2015, methodology): “Konidaris et al. (2011) introduced TD γ , a parameter-free alternative to TD( λ ) based on a multi-step update target 7.”
Policy Evaluation Using the Ω-Return (2015, methodology): “Recent work has begun to explore the statistical basis of complex estimates of return, and how we might reformulate them to be more statistically efficient [4].”