Interpolation-based Q-learning

By Csaba Szepesvari, William D. Smart. 2004

91 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Management Science and Operations Research

Citations in the corpus

Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “Furthermore, additional assumptions are not necessarily made on the value-function class (e.g., that b T F being a contraction (Gordon, 1995; Szepesv´ari & Smart, 2004; Lizotte, 2011; Pires & Szepesv´ari, 2016)), and can instead take the form of requiring another function class to realize other…”
Dynamic policy programming (2010, background): “Also, there are some studies in the literature concerning the asymptotic convergence of Q-learning in the presence of function approximation (Melo et al., 2008; Szepesvári and Smart, 2004).”
Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path (2006, methodology): “Q-learning with interpolative function approximation was considered by Szepesvari and Smart (2004) , who derived asymptotic convergence and performance bound guarantees.”
Multi-agent Reinforcement Learning: An Overview (2010, methodology): “Q-learning has been combined with Q-function approximators relying on, e.g., basis-function representations [110, 121 ], neural networks and self-organizing maps [127], and fuzzy rule bases [41, 47, 56].”
Model-based function approximation in reinforcement learning (2007, background): “For example, Interpolative Function Approximator based Q-Learning also builds upon Gordon’s convergence proofs for averagers, defining a modelfree algorithm that can converge to the optimal value function in the limit [17].”
A New Learning Algorithm for Optimal Stopping (2009, methodology): “i0=1 p(i0ji;u)V (i0)); ( 29 ) where (i;u) 0. Now we shall describe a gradient scheme for estimating the optimal V (i0) using linear function approximations for V and p . Let r2 Rm and q2Rn.”
Toward Off-Policy Learning Control with Function Approximation (2010, methodology): “The modifications to our analysis would be standard (Szepesvári & Smart (2004) used this condition in a similar context).”
Algorithms for Reinforcement Learning (2010, methodology): “Szepesvári and Smart (2004) prove that this algorithm converges almost surely as long as (i) the function-class Qθ satisfies the above interpolation property and the mapping θ 7→ Qθ is a non-expansion (i.e., ‖Qθ−Qθ′‖∞ ≤ ‖θ−θ′‖∞ holds for any θ, θ′ ∈ Rd); (ii) the local step-size sequences (αt,i; t…”
Non-delusional Q-learning and value-iteration (2018, background): “Numerous modifications of the basic update, restrictions on approximators, and training regimes have been proposed to ensure convergence or improve approximation error [12, 13, 27, 18, 17, 21].”
Experience Replay for Real-Time Reinforcement Learning Control (2012, methodology): “it does converge (under mild assumptions on the approximator), if the policy used to choose actions is fixed [20], [34].”
Q-learning with Nearest Neighbors (2018, methodology): “Specific work Method Continuous Unknown Single Online Non-asymptotic state space transition Kernel sample path update guarantees [10], [36], [37] Finite-state approximation Yes No No Yes Yes [43], [22], [41] Q-learning No Yes Yes Yes No [20], [3], [18] Q-learning No Yes Yes Yes Yes [23] Q-learning N”
Reinforcement Learning in Continuous State and Action Spaces (2012, background): “Furthermore,for specific functions— so-called averagers—Q-learning does converge (Gordon, 1995; Szepesvári and Smart, 2004).”
An analysis of reinforcement learning with function approximation (2008, methodology): “Sample-based methods (Ormoneit & Sen, 2002; Szepesvari & Smart, 2004 ) further generalize the applicability of soft-state aggregation methods by using spreading functions/kernels (Ribeiro & Szepesvari, 1996).”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, background): “Various modiﬁcations have been proposed to improve convergence or approximation error (Gordon, 1995; 1999; Szepesv´ari & Smart, 2004; Melo & Ribeiro, 2007; Maei et al., 2010; Munos et al., 2016); but it remains difﬁcult to reliably attain both robustness and scalability.”
Iterative Value-Aware Model Learning (2018, methodology): “The use of the Bellman [optimality] operator is central to value-based approaches such as the family of (Approximate) Value Iteration [Gordon, 1995, Szepesvári and Smart, 2004, Ernst et al., 2005, Munos and Szepesvári, 2008, Farahmand et al., 2009, Farahmand and Precup, 2012, Mnih et al., 2015,…”