Beyond the One Step Greedy Approach in Reinforcement Learning
By Yonathan Efroni, Gal Dalal, Bruno Scherrer, Shie Mannor. 2018
55 citations recorded by the source corpus.
The famous Policy Iteration algorithm alternates between policy improvement and policy evaluation. Implementations of this algorithm with several variants of the latter evaluation stage, e.g, $n$-step and trace-based returns, have been analyzed in previous works. However, the case of multiple-step lookahead policy improvement, despite the recent increase in empirical evidence of its strength, has to our knowledge not been carefully analyzed yet. In this work, we introduce the first such analysis. Namely, we formulate variants of multiple-step policy improvement, derive new algorithms using these definitions and prove their convergence. Moreover, we show that recent prominent Reinforcement Learning algorithms are, in fact, instances of our framework. We thus shed light on their empirical success and give a recipe for deriving new algorithms for future study.
Subfields: Artificial Intelligence, Management Science and Operations Research
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ching-An Cheng
- Xinyan Yan
- Byron Boots
- Sergey Levine
- Chelsea Finn
- Nan Jiang
- Sham M. Kakade
- Dieter Fox
In Management Science and Operations Research
Citations in the corpus
- Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “This biasvariance tradeoff can work well in practice, but can also diverge when not tuned carefully [14, 16, 9].”