TD γ : Re-evaluating Complex Backups in Temporal Difference Learning

By George Dimitri Konidaris, Philip S. Thomas. 2011

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Sergey Levine
2. Philip S. Thomas
3. Hado van Hasselt
4. Ofir Nachum
5. A. Mahmood
6. Scott Niekum
7. Emma Brunskill
8. Richard S. Sutton

Citations in the corpus

• Meta-Gradient Reinforcement Learning (2018, background): “Konidaris et al. [2011] derive a maximum-likelihood estimator, TD( γ ), that weights the n -step returns according to the discount factor, leading to a parameter-free algorithm for temporal-difference learning with linear function approximation.”
• Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning (2016, background): “Recent work has suggested that the λ-return could be replaced by more statistically principled complex returns like the γ-return (Konidaris et al., 2011) or Ωreturn (Thomas et al.”
• True Online Temporal-Difference Learning (2015, methodology): “Konidaris et al. (2011) introduced TD γ , a parameter-free alternative to TD( λ ) based on a multi-step update target 7.”
• Policy Evaluation Using the Ω-Return (2015, methodology): “[4] remedied this recently by showing that the λ-return is the maximum likelihood estimator of V (st) given three assumptions.”