A Unified Analysis of Value-Function-Based Reinforcement-Learning Algorithms

By Csaba Szepesvari, Michael L. Littman. 1999

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Learning to Act by Predicting the Future (2016, background): “While the understanding of the convergence of both types of methods has since improved (Szepesv´ari & Littman, 1999; Tsitsiklis, 2002; Even-Dar & Mansour, 2003), the argument for TD versus Monte Carlo is to this day empirical (Sutton & Barto, 2017).”
Mean Field Multi-Agent Reinforcement Learning (2018, positive): “See Theorem 1 in Jaakkola et al. (1994) and Corol-lary 5 Szepesvári & Littman (1999) for detailed derivation.”
A survey and critique of multiagent deep reinforcement learning (2018, background): “Q-learning is proven to converge to Q∗ if state and action spaces are discrete and finite, the sum of the learning rates goes to infinity (so that each state-action pair is visited infinitely often) and that the sum of the squares of the learning rates is finite (which is required to show that the c”
Nash Q-Learning for General-Sum Stochastic Games (2003, methodology): “Our proof relies on the following lemma by Szepesvári and Littman (1999), which establishes the convergence of a general Q-learning process updated by a pseudo-contraction operator.”
Algorithms for Reinforcement Learning (2010, methodology): “, 2008), learning and acting in games or under some other optimization criteria (Littman, 1994; Heger, 1994; Szepesvári and Littman, 1999; Borkar and Meyn, 2002), or the development of hierarchical and multi-time-scale methods (Dietterich, 1998; Sutton et al.”
Value Function Approximation in Zero-Sum Markov Games (2002, background): “(1) as a generalizedmax operation (Szepesvari & Littman, 1999) minimax-Q simply uses Eq.”
Certifiable Distributional Robustness with Principled Adversarial Training (2017, background): “In the former case, since the update (20) simply modiﬁes the state-action transitions (independent of Q ), standard results on convergence for tabular Q-learning (e.g. Szepesv´ari and Littman [47]) apply under these adversarial dynamics.”
Multiagent Reinforcement Learning for Multi-Robot Systems: A Survey (2004, background): “A complete convergence proof was provided in the works thereafter [34], [39], [42], which can be summarized in the following theorem:”
Interpolation-based Q-learning (2004, methodology): “4 The extension of Theorem 4 of (Szepesv´ari & Littman, 1999) to non-stationary processes can be obtained thanks to (cid:82) where f is any function satisfying f ∈ L 1 ( µ ).”
Nash Q-Learning for General-Sum Stochastic Games (2003, methodology): “Our proof relies on the following lemma by Szepesvári and Littman (1999), which establishes the convergence of a general Q-learning process updated by a pseudo-contraction operator.”