Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximation

By Hamid Reza Maei. 2018

36 citations recorded by the source corpus.

We present the first class of policy-gradient algorithms that work with both state-value and policy function-approximation, and are guaranteed to converge under off-policy training. Our solution targets problems in reinforcement learning where the action representation adds to the-curse-of-dimensionality; that is, with continuous or large action sets, thus making it infeasible to estimate state-action value functions (Q functions). Using state-value functions helps to lift the curse and as a result naturally turn our policy-gradient solution into classical Actor-Critic architecture whose Actor uses state-value function for the update. Our algorithms, Gradient Actor-Critic and Emphatic Actor-Critic, are derived based on the exact gradient of averaged state-value function objective and thus are guaranteed to converge to its optimal solution, while maintaining all the desirable properties of classical Actor-Critic methods with no additional hyper-parameters. To our knowledge, this is the first time that convergent off-policy learning methods have been extended to classical Actor-Critic methods with function approximation.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Generalized Off-Policy Actor-Critic (2019, methodology): “Second, we prove the Generalized Off-Policy Policy Gradient (GOPPG) Theorem, which gives the policy gradient of the counterfactual objective.”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, methodology): “2 This motivated further development, including a recent actor-critic algorithm proven to converge when the critic uses linear function approximation [Maei, 2018], as well as several methods using the approximate off-policy gradient such as Deterministic Policy Gradient (DPG) [Silver et al.”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, background): “Maei (2018) tries to show the convergence of a GTD critic under a slowly changing target policy with results from Konda (2002).”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “…for GEM-ETD(0) mainly because the magnitude of δ̄t in Eq (5) varies dramatically at different states, which makes the supervised learning of κ hard. dient only in a limiting sense under a fixed π (see Theorem 2 in Maei (2018)) and has potentially unbounded variance, similar to how Mt tracks mπ(St).”