Statistically Efficient Off-Policy Policy Gradients

By Nathan Kallus, Masatoshi Uehara. 2020

43 citations recorded by the source corpus.

Policy gradient methods in reinforcement learning update policy parameters by taking steps in the direction of an estimated gradient of policy value. In this paper, we consider the statistically efficient estimation of policy gradients from off-policy data, where the estimation is particularly non-trivial. We derive the asymptotic lower bound on the feasible mean-squared error in both Markov and non-Markov decision processes and show that existing estimators fail to achieve it in general settings. We propose a meta-algorithm that achieves the lower bound without any parametric assumptions and exhibits a unique 3-way double robustness property. We discuss how to estimate nuisances that the algorithm relies on. Finally, we establish guarantees on the rate at which we approach a stationary point when we take steps in the direction of our new estimated policy gradient.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Learning When-to-Treat Policies (2019, background): “…2019); Munos and Szepesvári (e.g., 2008)—this can be viewed as a similar analogue to the overlap assumption as in Assumption 3. policy gradient, and Kallus and Uehara (2020) provides regret bounds in the restricted setting where the policy value is a concave function of the policy parameters.”
Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning (2020, methodology): “) We use this to derive the efficiency bound for the NMDP case and compare to Kennedy (2019); Díaz et al.”