An Off-policy Policy Gradient Theorem Using Emphatic Weightings

By Ehsan Imani, Eric Graves, Martha White. 2018

76 citations recorded by the source corpus.

Policy gradient methods are widely used for control in reinforcement learning, particularly for the continuous action setting. There have been a host of theoretically sound algorithms proposed for the on-policy setting, due to the existence of the policy gradient theorem which provides a simplified form for the gradient. In off-policy learning, however, where the behaviour policy is not necessarily attempting to learn and follow the optimal policy for the given task, the existence of such a theorem has been elusive. In this work, we solve this open problem by providing the first off-policy policy gradient theorem. The key to the derivation is the use of $emphatic$ $weightings$. We develop a new actor-critic algorithm$\unicode{x2014}$called Actor Critic with Emphatic weightings (ACE)$\unicode{x2014}$that approximates the simplified gradients provided by the theorem. We demonstrate in a simple counterexample that previous off-policy policy gradient methods$\unicode{x2014}$particularly OffPAC and DPG$\unicode{x2014}$converge to the wrong solution whereas ACE finds the optimal solution.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “This may lead to suboptimal solutions in certain cases (see Imani et al. (2018) for some examples).”
Generalized Off-Policy Actor-Critic (2019, methodology): “Second, we prove the Generalized Off-Policy Policy Gradient (GOPPG) Theorem, which gives the policy gradient of the counterfactual objective.”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, methodology): “Many practical off-policy policy algorithms including DDPG (Silver et al., 2014), ACER (Wang et al., 2016), and Off-PAC with emphatic weightings (Imani et al., 2018) are based on the gradient expression in the Off-PAC algorithm (Degris et al., 2012).”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “Control: Off-policy actor-critic methods (Degris et al., 2012; Imani et al., 2018) aim to maximize the excursion objective J ( π ) .”
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning (2019, methodology): “For the actor portion of our algorithm, we need a slightly different emphatic weighting update than that in ETD(λ), corresponding to the update used in Imani et al. [2018]. Define”
Statistically Efficient Off-Policy Policy Gradients (2020, unknown): “Many of the existing oﬀ-policy RL algorithms including DDPG (Silver et al., 2014) and Oﬀ-PAC with emphatic weightings (Imani et al., 2018) also use the above trick, i.e. , ignoring the state distribution mismatch.”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “While there have been efforts to combine actor-critic algorithms with off-policy learning (Degris et al., 2012; Imani et al., 2018; Maei, 2018; Zhang et al., 2019; Liu et al., 2019), none of the resulting off-policy actor-critic algorithms is provably convergent under function approximation.”