Analysis of Some Incremental Variants of Policy Iteration: First Steps Toward Understanding Actor-Cr

By Ronald J. Williams, Leemon C. Baird III. 1993

Subfields: Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Generalized Markov Decision Processes: Dynamic-programming and Reinforcement-learning Algorithms (1996, background): “Most of the previous results on asynchronous policy iteration can be repeated since those proofs depend only on the monotonicity and the contraction properties of the involved operators [61, 40].”
Reinforcement Learning: A Tutorial Survey and Recent Advances (2009, unknown): “Some of the initial works in bounding the behavior of greedy policies obtained via function approximation are Williams and Baird (1993) and Singh and Yee (1994).”
Reinforcement learning in continuous time: advantage updating (1994, background): “Analysis has shown that many reasonablelooking algorithms for reinforcement learning can fail to converge in some situations (Williams and Baird, 1990, 1993).”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, background): “The direct method has no convergence guarantees, and indeed there exist several famous examples of divergence when used with classic RL targets [5, 65, 70], however its ubiquity in the RL community is testament to its ease of implementation and empirical success [42, 55].”
Approximate policy iteration: a survey and some new methods (2011, methodology): “However, even in that case, the convergence properties of optimistic policy iteration are fragile and depend on the implementation details, as discussed in [BeT96], Section 5.4; see also Williams and Baird [WiB93] who provide remarkable counterexamples to the convergence of a version of asynchronous”
Problem solving with reinforcement learning (1995, background): “These two elements are now more generally called the actor and the critic (thus AHC systems are often called Actor-Critic methods (Williams and Baird 1993a)).”