Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning

By Shixiang Shane Gu, Timothy P. Lillicrap, Zoubin Ghahramani, Richard E. Turner, Bernhard Schölkopf, Sergey Levine. 2017

179 citations recorded by the source corpus.

Off-policy model-free deep reinforcement learning methods using previously collected data can improve sample efficiency over on-policy policy gradient techniques. On the other hand, on-policy algorithms are often more stable and easier to use. This paper examines, both theoretically and empirically, approaches to merging on- and off-policy updates for deep reinforcement learning. Theoretical results show that off-policy updates with a value function estimator can be interpolated with on-policy policy gradient updates whilst still satisfying performance bounds. Our analysis uses control variate methods to produce a family of policy gradient algorithms, with several recently proposed algorithms being special cases of this family. We then provide an empirical comparison of these techniques with the remaining algorithmic details fixed, and show how different mixing of off-policy gradient estimates with on-policy samples contribute to improvements in empirical performance. The final algorithm provides a generalization and unification of existing deep policy gradient techniques, has theoretical guarantees on the bias introduced by off-policy updates, and improves on the state-of-the-art model-free deep RL methods on a number of OpenAI Gym continuous control benchmarks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, background): “Unfortunately, in recent reported results, it is not uncommon for the top-N trials to be selected from among several trials (Wu et al. 2017; Mnih et al. 2016) or averaged over only small number of trials ( N < 5 ) (Gu et al. 2017; Wu et al. 2017).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “Eﬀorts at combining the stability of trust region policy-based methods with the sample eﬃciency of value-based methods have focused on using oﬀ-policy data to better train a value estimate, which can be used as a control variate for variance reduction (Gu et al., 2017a,b).”
Experience Replay for Continual Learning (2018, background): “a via experience replay buffers is a common technique in RL. However, their introduction has primarily been driven by the goal of data-efﬁcient learning on single tasks [Lin, 1992, Mnih et al., 2015, Gu et al., 2017]. Research in this vein has included prioritized replay for maximizing the impact of”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “This approximate gradient is used as a starting point in many widely used deep reinforcement learning algorithms (Silver et al., 2014; Lillicrap et al., 2015; Wang et al., 2016; Gu et al., 2017b).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Because these methods directly estimate the gradient of the expected reward RL objective, they exhibit stable convergence both in theory and practice (Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015a; Gu et al., 2017a).”
MOPO: Model-based Offline Policy Optimization (2020, background): “While many off-policy RL algorithms [53, 11, 31, 48, 43, 27, 20, 24, 25] can in principle be applied to a batch offline setting, they perform poorly in practice [19].”
Generalized Off-Policy Actor-Critic (2019, methodology): “To this end, the Off-Policy Policy Gradient (OPPG) Theorem (Degris et al., 2012; Maei, 2018; Imani et al., 2018) was developed and has been widely used (Silver et al., 2014; Lillicrap et al., 2015; Wang et al., 2016; Gu et al., 2017; Ciosek and Whiteson, 2017; Espeholt et al., 2018).”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, methodology): “…critic uses linear function approximation [Maei, 2018], as well as several methods using the approximate off-policy gradient such as Deterministic Policy Gradient (DPG) [Silver et al., 2014, Lillicrap et al., 2015], ACER [Wang et al., 2016], and Interpolated Policy Gradient (IPG) [Gu et al., 2017].”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “…algorithms such as Deep Q-Network (Mnih et al., 2015, 2013) and Deep Deterministic Policy Gradients DDPG (Lillicrap et al., 2015) are biased (Gu et al., 2017) because behavior of past policies may be very different from that of the current policy and hence old data may not be a good…”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “It is tempting to apply various off-policy methods to the problem of learning from demonstration, such as policy gradient variants (Gu et al., 2017; 2016; Degris et al., 2012; Wang et al., 2016), Q-learning (Watkins & Dayan, 1992) and Retrace (Munos et al.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, methodology): “One possible option would also be to to use techniques similar to the ones employed in IPG (Gu et al., 2017).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “can be more data efficient, and hence there have been several attempts to merge the two [158, 94, 41, 39, 42].”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, methodology): “However, existing approaches to incorporating ofﬂine information into online policy gradients have shown limited beneﬁt (Gu et al., 2017b;a), in part due to the variance in gradients incurred due to incorporating off-policy data.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “[118] introduced the Interpolated Policy Gradient (IPG) algorithm and showed a connection between ACER and DDPG: they are a pair of reparametrization terms (they are special cases of IPG) when they are put under the same stochastic policy setting, and when the policy is deterministic they collapse i”
Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control (2017, methodology): “However, the architecture which we found to be the best (400,300) is not the one which is used in reporting results for baselines results in [10, 12].”
Evolution-Guided Policy Gradient in Reinforcement Learning (2018, background): “Notably, ERL is able to learn on the 3D quadruped locomotion Ant benchmark where DDPG normally fails to make any learning progress [13, 23, 24].”