Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic

By Shixiang Shane Gu, Timothy P. Lillicrap, Zoubin Ghahramani, Richard E. Turner, Sergey Levine. 2016

364 citations recorded by the source corpus.

© ICLR 2019 - Conference Track Proceedings. All rights reserved. Model-free deep reinforcement learning (RL) methods have been successful in a wide variety of simulated domains. However, a major obstacle facing deep RL in the real world is their high sample complexity. Batch policy gradient methods offer stable learning, but at the cost of high variance, which often requires large batches. TD-style methods, such as off-policy actor-critic and Q-learning, are more sample-efficient but biased, and often require costly hyperparameter sweeps to stabilize. In this work, we aim to develop methods that combine the stability of policy gradients with the efficiency of off-policy RL. We present Q-Prop, a policy gradient method that uses a Taylor expansion of the off-policy critic as a control variate. Q-Prop is both sample efficient and stable, and effectively combines the benefits of on-policy and off-policy methods. We analyze the connection between Q-Prop and existing model-free algorithms, and use control variate theory to derive two variants of Q-Prop with conservative and aggressive adaptation. We show that conservative Q-Prop provides substantial gains in sample efficiency over trust region policy optimization (TRPO) with generalized advantage estimation (GAE), and improves stability over deep deterministic policy gradient (DDPG), the state-of-the-art on-policy and off-policy methods, on OpenAI Gym's MuJoCo continuous control environments.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Constrained Policy Optimization (2017, methodology): “, 2016) and have shown state-of-the-art performance in the deep RL setting (Duan et al., 2016; Gu et al., 2017), making the approach appealing for developing policy search methods for CMDPs.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “There have been efforts to increase the sample efﬁciency while retaining robustness by incorporating off-policy samples and by using higher order variance reduction techniques (O’Donoghue et al., 2016; Gu et al., 2016).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “Eﬀorts at combining the stability of trust region policy-based methods with the sample eﬃciency of value-based methods have focused on using oﬀ-policy data to better train a value estimate, which can be used as a control variate for variance reduction (Gu et al., 2017a,b).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Hence, current methods remain potentially unstable and require specialized algorithmic and theoretical development as well as delicate tuning to be effective in practice [10, 41, 8].”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “Other beneﬁts include robustness in the face of uncertain dynamics (Ziebart, 2010), imitation learning (Ziebart et al., 2008), and improved convergence and computational properties (Gu et al., 2016a).”
Neural Approaches to Conversational AI (2018, neutral): “Interested readers can refer to a few related works and the references therein for further details (Kakade, 2001; Peters et al., 2005; Schulman et al., 2015a,b; Mnih et al., 2016; Gu et al., 2017; Dai et al., 2018a; Liu et al., 2018b).”
Learning Combinatorial Optimization Algorithms over Graphs (2017, background): “It is known that off-policy reinforcement learning algorithms such as Q-learning can be more sample efficient than their policy gradient counterparts (Gu et al., 2016).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Because these methods directly estimate the gradient of the expected reward RL objective, they exhibit stable convergence both in theory and practice (Sutton et al., 2000; Kakade, 2002; Schulman et al., 2015a; Gu et al., 2017a).”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “Our analysis examines how Q-Prop (Gu et al., 2017), PGQ (O’Donoghue et al., 2017), and ACER (Wang et al., 2017), which are all recent works that combine on-policy with off-policy learning, are connected to each other (see Table 1).”
MOPO: Model-based Offline Policy Optimization (2020, background): “While many off-policy RL algorithms [52, 10, 30, 47, 42, 26, 19, 23, 24] can in principle be applied to a batch offline setting, they perform poorly in practice [18] due to poor extrapolation to out-of-distribution actions.”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “(2019) noticed that the previous variance reduction methods in PG overlooked the correlation across the times steps and ignored the randomness in the future steps (e.g., Grathwohl et al., 2018; Gu et al., 2017; Liu et al., 2018; Wu et al., 2018).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “In practice, it has been observed that these CVs indeed accelerate policy optimization, especially in simulated robot control tasks [20, 21, 23, 25, 36].”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “…[Witten, 1977, Barto et al., 1983] and many techniques since proposed to reduce variance of the estimates of this gradient [Konda and Tsitsiklis, 2000, Weaver and Tao, 2001, Greensmith et al., 2004, Peters et al., 2005, Bhatnagar et al., 2008, 2009, Grondman et al., 2012, Gu et al., 2016].”
Reward-estimation variance elimination in sequential decision processes (2018, background): “More recently state-action-dependent baselines have been investigated [2, 4, 8, 20].”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, background): “…RL. Off-policy reinforcement learning There is a large body of prior works in the domain of off-policy RL, including extensions of policy gradient [Gu et al., 2016, Degris et al., 2012, Wang et al., 2016] or Q-learning [Watkins and Dayan, 1992, Haarnoja et al., 2018, Munos et al., 2016].”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “NAF (Gu et al., 2016b) and ICNN (Amos et al., 2017) directly constrain the function family of Q-functions such that the optimization can be closed-form or tractable.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “This avoids training a separate deterministic policy (unlike Q-Prop) for the off-policy part and encourages on-policy exploration and an implicit trust region update.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “For example see the entropy regularization terms used in Mnih et al. (2016) or the KL constraints employed by work on trust-region based methods (Schul-man et al., 2015; 2017b; Gu et al., 2017; Wang et al., 2017).”