An Analysis of Temporal-Difference Learning with Function Approximation

By John N. Tsitsiklis, Benjamin Van Roy. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, background): “Note that when learning by experience replay, it is necessary to learn off-policy (because our current parameters are different to those used to generate the sample), which motivates the choice of Q-learning.”
Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “Temporal difference learning methods, such as Q-learning, have been known to diverge when used with nonlinear function approximators [Tsitsiklis and Roy, 1997].”
Compatible Value Gradients for Reinforcement Learning of Continuous Deep Policies (2015, background): “The convergence properties of TD-learning and related algorithms have been studied extensively, see (Tsitsiklis and Roy, 1997; Dann et al., 2014).”
Massively Parallel Methods for Deep Reinforcement Learning (2015, background): “However, it is well-known that the Q-learning algorithm is highly unstable when combined with non-linear function approximators such as deep neural networks (Tsitsiklis & Roy, 1997).”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “If seen in isolation, one might perhaps be tempted to think the observed instability is related to inherent instability problems of off-policy learning with function approximation (Baird, 1995; Tsitsiklis and Van Roy, 1997; Sutton et al., 2008; Maei, 2011; Sutton et al., 2015).”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “Prior work on Q-learning and stochastic approximation analyzes time-varying μ, but either without function approximation (Watkins and Dayan, 1992; Tsitsiklis, 1994; Devraj and Meyn, 2017), or when fully online (Tsitsiklis and Roy, 1997), unlike our setting, that uses replay buffer data.”
Deep Reinforcement Learning and the Deadly Triad (2018, methodology): “This reproduces at scale the pattern seen for the modiﬁed Tsitsiklis and Van Roy (1997) example, when more learnable parameters are added to the canonical formulation: values initially grow very large, but then return to more accurate value estimates.”
Deep Reinforcement Learning for Automated Radiation Adaptation in Lung Cancer (2017, background): “A linear function approximator was first considered in early literature [23] for computational reasons.”