Policy Gradient Methods for Off-policy Control
By Lucas Lehnert, Doina Precup. 2015
4 citations recorded by the source corpus.
Off-policy learning refers to the problem of learning the value function of a way of behaving, or policy, while following a different policy. Gradient-based off-policy learning algorithms, such as GTD and TDC/GQ, converge even when using function approximation and incremental updates. However, they have been developed for the case of a fixed behavior policy. In control problems, one would like to adapt the behavior policy over time to become more greedy with respect to the existing value function. In this paper, we present the first gradient-based learning algorithms for this problem, which rely on the framework of policy gradient in order to modify the behavior policy. We present derivations of the algorithms, a convergence theorem, and empirical evidence showing that they compare favorably to existing approaches.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Management Science and Operations Research, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Pratik Chaudhari
- Rasool Fakoor
- Alex Smola
- Arthur Gretton
- Bernhard Schölkopf
- Kenji Fukumizu
- Francis R. Bach
- Sergey Levine
In Computational Theory and Mathematics
In Management Science and Operations Research
In Reinforcement Learning
Citations in the corpus
- P3O: Policy-on Policy-off Policy Optimization (2019, background): “Let us note there are a number of works that learn action value functions using off-policy data, e.g.,Wang et al. (2013); Hausknecht and Stone (2016); Lehnert and Precup (2015) that achieve varying degrees of success on reinforcement learning benchmarks.”