A short variational proof of equivalence between policy gradients and soft Q learning

By Pierre H. Richemond, Brendan Maginnis. 2017

6 citations recorded by the source corpus.

Two main families of reinforcement learning algorithms, Q-learning and policy gradients, have recently been proven to be equivalent when using a softmax relaxation on one part, and an entropic regularization on the other. We relate this result to the well-known convex duality of Shannon entropy and the softmax function. Such a result is also known as the Donsker-Varadhan formula. This provides a short proof of the equivalence. We then interpret this duality further, and use ideas of convex analysis to prove a new policy inequality relative to soft Q-learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

Citations in the corpus

• Variational Bayesian Reinforcement Learning with Regret Bounds (2018, neutral): “The update rule we derive is similar to that used in ‘soft’ Q-learning (so-called since the ‘hard’ max is replaced with a soft-max) [6, 16, 18, 30, 47].”