Reinforcement Learning with Function Approximation Converges to a Region

By Geoffrey J. Gordon. 2000

153 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Reinforcement Learning for RoboCup Soccer Keepaway (2005, methodology): “This method is not guaranteed to converge to the optimal policy in all cases, but several lines of evidence suggest that it is near a good solution (Gordon, 2001; Tsitsiklis & Van Roy, 1997; Sutton, 1996) and recent results show that it does indeed con- verge, as long as the action-selection policy…”
A Convergent Form of Approximate Policy Iteration (2002, background): “Arguably, the best successes of the ﬁeld were generated in this way (e.g., [15]), and there are a few positive convergence results, particularly for the case of linear approximators [16, 7, 8].”
Scaling Reinforcement Learning toward RoboCup Soccer (2001, methodology): “This method is known not to converge in the conventional sense, but several lines of evidence suggest that it nevertheless remains near a good solution (Gordon, 2001; Tsitsiklis & Van Roy, 1997; Sutton, 1996).”
On the sample complexity of reinforcement learning. (2003, methodology): “There are a plethora of greedy approximate methods in the liter ature (see Sutton and Barto [1998], Bertsekas and Tsitsiklis [1996], and Gordon [1999]).”
A New Softmax Operator for Reinforcement Learning (2016, methodology): “We evaluated SARSA with various policies on the multi-passenger taxi domain introduced by (Dearden et al., 1998).”
Policy Gradient Methods for Off-policy Control (2015, background): “SARSA also suﬀers from this problem and is only guaranteed to converge to a sub-space of policies [4, 3].”
Reinforcement learning for factored Markov decision processes (2002, background): “At best, the linear approximation has been shown to converge to a bounded region for the SARSA update rule [Gordon 2001].”
Particle swarm optimization for generating interpretable fuzzy reinforcement learning policies (2016, methodology): “ence, an optimal policy is given after nishing the algorithm in iteration Kby evaluating ˇ[x K](s) 2 argmax a Q[x K](s;a). In practice function approximation in RL is not known to converge to a point [13] and is prone to overestimation of utility values [34]. Since neural networks are not averagers,”
State of the Art Control of Atari Games Using Shallow Reinforcement Learning (2015, methodology): “Theoretical results suggest that, as an on-policy method, Sarsa(λ) may be more stable in the linear function approximation case than off-policy methods such as Q-learning, which are known to risk divergence [1, 10, 18].”
Reinforcement Learning: An Introduction (1998, background): “Linear semi-gradient Sarsa with "-greedy action selection does not converge in the usual sense, but does enter a bounded region near the best solution (Gordon, 1996a, 2001).”