P3O: Policy-on Policy-off Policy Optimization

By Rasool Fakoor, Pratik Chaudhari, Alex Smola. 2019

66 citations recorded by the source corpus.

On-policy reinforcement learning (RL) algorithms have high sample complexity while off-policy algorithms are difficult to tune. Merging the two holds the promise to develop efficient algorithms that generalize across diverse environments. It is however challenging in practice to find suitable hyper-parameters that govern this trade off. This paper develops a simple algorithm named P3O that interleaves off-policy updates with on-policy updates. P3O uses the effective sample size between the behavior policy and the target policy to control how far they can be from each other and does not introduce any additional hyper-parameters. Extensive experiments on the Atari-2600 and MuJoCo benchmark suites show that this simple technique is effective in reducing the sample complexity of state-of-the-art algorithms. Code to reproduce experiments in this paper is at this https URL.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Accelerating Online Reinforcement Learning with Offline Datasets (2020, positive): “Offline RL methods aim to address this issue, often by constraining the actor updates to avoid excessive deviation from the data distribution [24, 43, 15, 16, 14, 2, 23, 10, 8, 28, 40, 25, 51].”
MOReL : Model-Based Offline Reinforcement Learning (2020, methodology): “A number of algorithms based on MPC [23, 64], search-based planning [65, 25], dynamic programming [49, 26], or policy optimization [27, 51, 66, 67] can be used to approximately realize this.”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “…the off-policy updates of Silver et al. (2014) and temporal difference value-updates using (3), we emphasize that the general idea behind CDC can be utilized with other forms of actorcritic updates such as those considered by Haarnoja et al. (2018); Fakoor et al. (2019); Fujimoto et al. (2018).”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “…if they accumulate data in a buffer D , i.e. experience replay (Lin, 1992), they are off-policy (Mnih et al., 2015; Lillicrap et al., 2016; Gu et al., 2016; 2017b; Haarnoja et al., 2018; Fujimoto et al., 2019; Fakoor et al., 2019) because not all the data in buffer comes from the current policy.”
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning (2020, methodology): “This method of computing propensity score, or the importance ratio, is close to two-sample tests [3, 15, 16] in the statistics literature.”
Generalized Proximal Policy Optimization with Sample Reuse (2021, background): “Other approaches have combined on-policy and off-policy policy gradients, with the goal of balancing the variance of on-policy methods and the bias of off-policy methods [7, 9, 10, 16, 24].”
Meta-Q-Learning (2019, methodology): “The coefﬁcient of the proximal term in the adaptation-phase objective (18– 19) using the effective sample size (ESS) is inspired from the recent work of Fakoor et al. (2019).”
Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “To solve these issues, offline RL is an excellent choice to deal with practical problems [3, 24, 35, 42, 15, 28, 4, 23, 54, 12], aiming at learning from a fixed dataset without interaction with environments.”