Generalized Off-Policy Actor-Critic

By Shangtong Zhang, Wendelin Böhmer, Shimon Whiteson. 2019

45 citations recorded by the source corpus.

We propose a new objective, the counterfactual objective, unifying existing objectives for off-policy policy gradient algorithms in the continuing reinforcement learning (RL) setting. Compared to the commonly used excursion objective, which can be misleading about the performance of the target policy when deployed, our new objective better predicts such performance. We prove the Generalized Off-Policy Policy Gradient Theorem to compute the policy gradient of the counterfactual objective and use an emphatic approach to get an unbiased sample from this policy gradient, yielding the Generalized Off-Policy Actor-Critic (Geoff-PAC) algorithm. We demonstrate the merits of Geoff-PAC over existing algorithms in Mujoco robot simulation tasks, the first empirical success of emphatic algorithms in prevailing deep RL benchmarks.

Subfields: Artificial Intelligence, Software, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. George Tucker
4. Pieter Abbeel
5. Aviral Kumar
6. Ofir Nachum
7. Shixiang Shane Gu
8. Justin Fu

In Software

1. Sergey Levine
2. George Tucker
3. Lihong Li
4. Aviral Kumar
5. Alekh Agarwal
6. Bo Liu
7. Hengshuai Yao
8. Qiang Liu

In Reinforcement Learning

1. Sergey Levine
2. Aviral Kumar
3. George Tucker
4. Ofir Nachum
5. Chelsea Finn
6. Justin Fu
7. Pieter Abbeel
8. Shixiang Shane Gu

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Imani et al. (2018); Zhang et al. (2019) show that similar methods can be used to estimate the off-policy policy gradient and thus be used in an off-policy actor critic method.”
• Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [21, 6, 27, 13, 9, 5, 34, 52, 48, 4].”
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “Its accuracy can be improved by various techniques [6,11,16,36].”
• Off-Policy Policy Gradient with Stationary Distribution Correction (2019, methodology): “Finally, we present convergence results quantifying the error in our critic and propensity score computations while Zhang et al. (2019) assume a perfect oracle for both and rely on a truly unbiased gradient estimator for the convergence results.”
• Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “5 Imani et al. (2018) approximate m π ( S t ) with the followon trace M t , yielding the ACE update | ] exists and π is fixed, Sutton et al. (2016) show that lim t →∞ µ [ M t | S t = s ] = m π ( s ) . established The existence of this limit is later in Lemma 1 in Zhang et al. (2019).”
• Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “We, therefore, refer the reader to Zhang et al. (2019) for full details.”