Toward Off-Policy Learning Control with Function Approximation

By Hamid Reza Maei, Csaba Szepesvari, Shalabh Bhatnagar, Richard S. Sutton. 2010

264 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Playing Atari with Deep Reinforcement Learning (2013, methodology): “en to converge when evaluating a ﬁxed policy with a nonlinear function approximator [14]; or when learning a control policy with linear function approximation using a restricted variant of Q-learning [15]. However, these methods have not yet been extended to nonlinear control. Perhaps the most simil”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “This is different from value function or policy learning from off-policy data (e.g., Precup et al., 2001; Maei et al., 2010; Sutton et al., 2016; Munos et al., 2016; Metelli et al., 2018), where the major goal is to ensure stability and convergence.”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “On-policy distributions (Tsitsiklis & Van Roy, 1997) and ﬁxed behavior distributions (Sutton et al., 2009b; Maei et al., 2010) have often been targeted for theoretical convergence analysis, and many works use importance sampling to correct for off-policyness (Precup et al., 2001; Munos et al.,…”
An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning (2015, methodology): “Parametric temporal-difference learning was first studied as the key “learning by generalization” algorithm in Samuel’s (1959) checker player.”
Linear Off-Policy Actor-Critic (2012, methodology): “…d b ( s ) on its diagonal; Π is a projection operator that projects a value function to the nearest representable value function given the function approximator; and T λ,γπ is the λ -weighted Bellman operator for the target policy π with termination probability γ (e.g., see Maei & Sutton, 2010).”
Dynamic policy programming (2010, neutral): “The gradient update ensures that PGAC asymptotically converges to a local maximum, given that an unbiased estimate of the gradient is provided by the critic (Maei et al., 2010; Bhatnagar et al., 2009; Konda and Tsitsiklis, 2003; Kakade, 2001).”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “Furthermore, the solution found by our algorithm minimizes the projected Bellman error, as in prior works when specialized to the same settings (Sutton et al., 2009; Maei et al., 2010; Liu et al., 2015; Macua et al., 2015).”
Reinforcement Learning in Continuous State and Action Spaces (2012, methodology): “The methods can be extended to control problems (Maei et al, 2010) with a greedy non-stationary estimation policy, although it is not yet clear how well the resulting Greedy-GQ algorithm performs in practice.”
Scalable Bilinear π Learning Using State and Action Features (2018, background): “The Greedy-GQ is an interesting control algorithm with linear approximation that has convergence guarantees under a relatively mild assumption (Maei et al., 2010).”
Deterministic Policy Gradient Algorithms (2014, methodology): “Speciﬁcally, we use gradient Q-learning in the critic (Maei et al., 2010), and note that under suitable conditions on the step-sizes, α θ , α w , α u , to ensure that the critic is updated on a faster time-scale than the actor, the critic will converge to the parameters minimising the MSPBE (Sutton…”
Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation (2018, methodology): “Temporal-Difference (TD) Learning methods solve this problem but they have been used for value function based methods (Sutton et al., 2009; Maei et al., 2010; Maei, 2011; Sutton et al., 2016).”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, background): “Various modiﬁcations have been proposed to improve convergence or approximation error (Gordon, 1995; 1999; Szepesv´ari & Smart, 2004; Melo & Ribeiro, 2007; Maei et al., 2010; Munos et al., 2016); but it remains difﬁcult to reliably attain both robustness and scalability.”
Algorithms for Reinforcement Learning (2010, background): “More recently, these algorithms have been extended to nonlinear function approximation (Maei et al., 2010a).”
Bayes-Adaptive Simulation-based Search with Value Function Approximation (2014, background): “Finally, more general updates such as gradient Q-learning could be applied with corresponding convergence guarantees [14].”
Efficient Sequential Decision Making (2017, background): “2, Chapter 6], and more recent papers [49, 48, 33, 34].”