Off-Policy Policy Gradient with Stationary Distribution Correction

By Yao Liu, Adith Swaminathan, Alekh Agarwal, Emma Brunskill. 2019

70 citations recorded by the source corpus.

We study the problem of off-policy policy optimization in Markov decision processes, and develop a novel off-policy policy gradient method. Prior off-policy policy gradient approaches have generally ignored the mismatch between the distribution of states visited under the behavior policy used to collect data, and what would be the distribution of states under the learned policy. Here we build on recent progress for estimating the ratio of the state distributions under behavior and evaluation policies for policy evaluation, and present an off-policy policy gradient optimization technique that can account for this mismatch in distributions. We present an illustrative example of why this is important and a theoretical convergence guarantee for our approach. Empirically, we compare our method in simulations to several strong baselines which do not correct for this mismatch, significantly improving in the quality of the policy discovered.

Subfields: Economics and Econometrics, Renewable Energy, Sustainability and the Environment, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

1. Bo Dai
2. Mohammad Norouzi
3. Yi-Hsun Su
4. Jiawei Huang
5. Thorsten Joachims
6. Dale Schuurmans
7. Yinlam Chow
8. Ilya Kostrikov

In Renewable Energy, Sustainability and the Environment

1. Jiawei Huang
2. Nan Jiang
3. Ofir Nachum
4. Sergey Levine

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Nan Jiang
4. George Tucker
5. Bo Dai
6. Lihong Li
7. Ilya Kostrikov
8. Aviral Kumar

Citations in the corpus

• Off-policy Bandits with Deficient Support (2020, methodology): “First, our conservative extrapolation method is related to the method proposed by [19].”
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, methodology): “Off-policy estimation and evaluation can also be used as a component for policy optimization (e.g., Jiang & Li, 2016; Gelada & Bellemare, 2019; Liu et al., 2019; Zhang et al., 2019).”
• AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “A more recent work (Liu et al., 2019) attempts to weight updates by estimated state-action distribution corrections.”
• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, background): “It also serves as a key algorithmic component of off-policy policy optimization (e.g., Dudík et al., 2011; Jiang & Li, 2016; Thomas & Brunskill, 2016; Liu et al., 2019).”
• EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “Furthermore, due to the dynamics of the MDP, taking out-of-distribution actions may lead to states not covered in the offline data, creating a snowball effect (Ross et al., 2011).”
• Minimax Confidence Interval for Off-Policy Evaluation and Policy Optimization (2020, methodology): “[9] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill.”
• An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “In our offline setup, we only assume access to samples from the behavior policy and focus on Q-learning methods without any form of importance correction, as opposed to (Swaminathan & Joachims, 2015; Liu et al., 2019).”