Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation

2009

Citations in the corpus

Reinforcement Learning via Fenchel-Rockafellar Duality (2020, neutral): “Typically, the Q -values are estimated by ﬁnding the ﬁxed point of the Bellman recurrence (Sutton et al., 2008, 2009; Scherrer, 2010; Liu et al., 2015; Dai et al., 2016; Du et al., 2017); i.e. , minimizing the (surrogate) squared diﬀerence between the LHS and RHS of (23), potentially with target…”
Least Squares Temporal Difference Methods: An Analysis under General Conditions (2012, neutral): “(In the case λ = 0, there are several convergent gradient-based oﬀ-policy TD variants; see Sutton et al. [33] and the references therein.)”
Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view (2010, positive): “…matrix inverse below exists1, it can be proved2 that v̂TD = ΦwTD with wTD = (Φ ′ΞLΦ)−1Φ′Ξr (1) As pointed out by Antos et al. (2008); Farahmand et al. (2008); Sutton et al. (2009), when the inverse exists, the above computation is equivalent to minimizing for v̂ ∈ span (Φ) the TD error ETD(v̂) :=…”
Nonparametric Return Distribution Approximation for Reinforcement Learning (2010, unknown): “1 shows the settings of this MDP, which is based on (Sutton & Barto, 1998) and (Sutton et al., 2009).”
Q-learning and enhanced policy iteration in discounted dynamic programming (2010, unknown): “[SMP09]).”
A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning (2016, unknown): “This algorithm, however, does not utilize eligibility traces and converges to a diﬀerent solution than TD meth-ods when function approximation is used [24].”
Linear Programming for Large-Scale Markov Decision Problems (2014, unknown): “For a survey on theoretical results for ADP see (Bertsekas and Tsitsiklis, 1996, Sutton and Barto, 1998), (Bertsekas, 2007, Vol. 2, Chapter 6), and more recent papers (Sutton et al., 2009b,a, Maei et al., 2009, 2010).”