Analysis of temporal-difference learning with function approximation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “For instance, it is well known that Q-learning can be unstable or even divergent when neural networks are used to parameterize optimal action-value functions [2, 9, 58].”
Generalized Off-Policy Actor-Critic (2019, background): “Tsitsiklis and Van Roy (1997) prove the convergence of on-policy linear TD.”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “The DQN addressed the fundamental instability problem of using function approximation in RL [83] by the use of two techniques: experience replay [45] and target networks.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Additionally, convergence results for reinforcement learning using function approximation are still scarce [74, 81, 82, 83, 80]; in general, stronger convergence guarantees are available for policy-gradient methods [55] than for value-based methods [20].”
Natural Gradient Deep Q-learning (2018, background): “In fact, it has been shown that TD algorithms can sometimes be divergent [Tsitsiklis and Van Roy, 1997].”
Reinforcement learning in robotic applications: a comprehensive survey (2021, background): “Conditions for convergence and bound on error is presented in (Tsitsiklis and Van Roy 1997).”
Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms (2019, unknown): “1), named temporal difference (TD) learning [58, 59, 60].”