Equivalence Between Policy Gradients and Soft Q-Learning

By John Schulman, Pieter Abbeel, Xi Chen. 2017

412 citations recorded by the source corpus.

Two of the leading approaches for model-free reinforcement learning are policy gradient methods and $Q$-learning methods. $Q$-learning methods can be effective and sample-efficient when they work, however, it is not well-understood why they work, since empirically, the $Q$-values they estimate are very inaccurate. A partial explanation may be that $Q$-learning methods are secretly implementing policy gradient updates: we show that there is a precise equivalence between $Q$-learning and policy gradient methods in the setting of entropy-regularized reinforcement learning, that "soft" (entropy-regularized) $Q$-learning is exactly equivalent to a policy gradient method. We also point out a connection between $Q$-learning methods and natural policy gradient methods. Experimentally, we explore the entropy-regularized versions of $Q$-learning and policy gradients, and we find them to perform as well as (or slightly better than) the standard variants on the Atari benchmark. We also show that the equivalence holds in practical settings by constructing a $Q$-learning method that closely matches the learning dynamics of A3C without using a target network or $\epsilon$-greedy exploration schedule.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “This will give rise to several prototypical methods that mirror corresponding techniques in standard reinforcement learning: policy gradients, actor-critic algorithms, and Q-learning.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “From this result, we can formulate a soft actor-critic algorithm, and we empirically show that it outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy PPO algorithm.”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “Even if one replaces the constraint on the average KL-divergence of TRPO with a simple regularization penalty (as in proximal policy gradient methods (Schulman et al., 2017b; Wang et al., 2016)), optimizing the resulting objective requires computing the gradient of the KL-divergence.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, background): “In some cases entropy regularization is expressed in the form of relative entropy [4, 6, 7, 31], and in other cases it is the standard entropy [47].”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “Similar derivation is presented in a concurrent work (Schulman et al., 2017).”
Self-Imitation Learning (2018, methodology): “The result in Figure 5 shows that our self-imitation learning improves PPO on Swimmer, Walker2d, and Ant tasks.”
Generalized Off-Policy Actor-Critic (2019, background): “This is a common policy parameterization for continuous-action problems (Schulman et al., 2015, 2017b).”
Smoothed Dual Embedding Control (2017, methodology): “k (Brockman et al., 2016) using the MuJoCo simulator (Todorov et al., 2012), comparing with trust region policy optimization (TRPO) (Schulman et al., 2015) and the proximal policy optimization (PPO) (Schulman et al., 2017b). Since the TRPO and PPO are only applicable for on-policy setting, for fairn”
Towards Characterizing Divergence in Deep Q-Learning (2019, background): “A related observation was made by (Schulman et al., 2017), who showed that a natural policy gradient could be viewed as approximately applying a version of Eq 12 for entropy-regularized Q-learning.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “As a ﬁrst observation, we can see that MPO gives stable learning on all tasks and, thanks to its fully off-policy implementation, is signiﬁcantly more sample efﬁcient than the on-policy PPO base-line.”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “(Haarnoja et al., 2017b) and (Schulman et al., 2017) propose maximum entropy RL methods to learn from environments.”
Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research (2018, background): “Are there other ways to stabilize training which does not limit the learning speed (e.g. clipped objective similar to the one used in PPO (Schulman et al., 2017b))?”
Diversity is All You Need: Learning Skills without a Reward Function (2018, methodology): “Related work has also examined connections between RL and information theory (Ziebart et al., 2008; Schulman et al., 2017; Nachum et al., 2017; Haarnoja et al., 2017) and developed maximum entropy algorithms with these ideas Haarnoja et al. (2018; 2017).”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “…for the poor sample efﬁciency of deep RL methods is on-policy learning: some of the most commonly used deep RL algorithms, such as TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017b) or A3C (Mnih et al., 2016), require new samples to be collected for (nearly) every update to the policy.”
Distral: Robust multitask reinforcement learning (2017, methodology): “KL divergence vs entropy regularization : With α = 0 , we get a purely entropy-regularized objective which does not couple and transfer across tasks [16, 24].”
A Brief Survey of Deep Reinforcement Learning (2017, positive): “Meanwhile, the A3C-based agent of Mirowski et al. [82] was additionally trained to construct a depth map given RGB inputs, which helps it in its task of learning to navigate a 3D environment.”
Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “In practical policy gradient, even though the optimal policy, given Markovian assumption, needs to be deterministic, the policy regularization is a dominant approach to make the policy stochastic and preserve the exploration thorough the stochasticity of the policy [51], [52].”
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018, methodology): “…the state decoder with backpropagation with the standard VAE loss and training the policy decoder by initializing with behavior cloning and doing RL ﬁnetuning with the reward function described above using PPO (Schulman et al., 2017b), backpropagating gradients into the encoder in both cases.”