PGQ: Combining policy gradient and Q-learning

By Brendan O'Donoghue, R´emi Munos, Koray Kavukcuoglu, Volodymyr Mnih. 2016

146 citations recorded by the source corpus.

Policy gradient is an efficient technique for improving a policy in a reinforcement learning setting. However, vanilla online variants are on-policy only and not able to take advantage of off-policy data. In this paper we describe a new technique that combines policy gradient with off-policy Q-learning, drawing experience from a replay buffer. This is motivated by making a connection between the fixed points of the regularized policy gradient algorithm and the Q-values. This connection allows us to estimate the Q-values from the action preferences of the policy, to which we apply Q-learning updates. We refer to the new technique as 'PGQL', for policy gradient and Q-learning. We also establish an equivalency between action-value fitting techniques and actor-critic algorithms, showing that regularized policy gradient techniques can be interpreted as advantage function learning algorithms. We conclude with some numerical examples that demonstrate improved data efficiency and stability of PGQL. In particular, we tested PGQL on the full suite of Atari games and achieved performance exceeding that of both asynchronous advantage actor-critic (A3C) and Q-learning.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “This technique was proposed as early as the first work on the REINFORCE algorithm (Williams and Peng, 1991; Williams, 1992), and is often used in recent methods (see, e.g., discussion by O’Donoghue et al. (2017)).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Although these approaches have typically been considered distinct, recent work (Norouzi et al., 2016; O’Donoghue et al., 2017) has begun to relate the two by posing entropy regularized expected reward as the target objec-tive (Williams & Peng, 1991).”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “The principal contribution of this work is a tractable, efﬁcient algorithm for optimizing arbitrary multimodal stochastic policies represented by energy-based models, as well as a discussion that relates this method to other recent algorithms in RL and probabilistic inference.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Related previous work on off-policy RL include (Precup et al., 2000; 2001; Wawrzynski, 2009; Geist & Scherrer, 2014; O’Donoghue et al., 2017) and (Harutyunyan et al.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “We have not considered purely policy-based RL algorithms such as trust-region policy optimisa-DQN tion (Schulman et al. 2015), nor actor-critic methods (Mnih et al. 2016; O’Donoghue et al. 2016).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “In this direction, several recent methods have successfully mixed high-variance, unbiased on-policy gradient estimates directly with low-variance, biased off-policy or model-based gradient estimates to improve performance (O’Donoghue et al., 2016; Wang et al., 2016b; Gu et al., 2017a).”
The Uncertainty Bellman Equation and Exploration (2017, neutral): “This is in contrast to the commonly used -greedy [20] and Boltzmann exploration strategies [19, 23] which simply inject noise into the agents actions.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “The PGQL algorithm (O’Donoghue et al., 2016a) uses an estimate of the action value function of the target policy to combine on-policy updates with those obtained from the Bellman error objective.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “A perspective that can also be extended to hierarchical policies (Florensa et al., 2017), and has recently been used to establish connections between Q-learning and policy gradient methods (O’Donoghue et al., 2016; Schulman et al., 2017a).”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al.”
A Brief Survey of Deep Reinforcement Learning (2017, positive): “Meanwhile, the A3C-based agent of Mirowski et al. [82] was additionally trained to construct a depth map given RGB inputs, which helps it in its task of learning to navigate a 3D environment.”
A unified view of entropy-regularized Markov decision processes (2017, methodology): “O’Donoghue et al. (2017, Section 3.1) study the stationary points of the objec-tive (16) and, similarly to us, show a connection between a certain type of value function and a policy achieving the stationary point.”
Evolved Policy Gradients (2018, background): “Prior works add auxiliary reward terms that emphasize exploration (4; 12; 21; 40; 2; 22) and entropy loss terms (20; 29; 9; 15).”
Learning to Walk via Deep Reinforcement Learning (2018, background): “However, the weight on the entropy term (“temperature”) is typically chosen heuristically [15, 17, 32, 33, 40].”
Recall Traces: Backtracking Models for Efficient Reinforcement Learning (2018, background): “, 2017), PGQ (O’Donoghue et al., 2016) and ACER (Wang et al.”