A unified view of entropy-regularized Markov decision processes

By Gergely Neu, Anders Jonsson, Vicenç Gómez. 2017

318 citations recorded by the source corpus.

We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Information Theory, Game & Decision Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Pieter Abbeel
  4. Mohit Bansal
  5. Percy Liang
  6. Jianfeng Gao
  7. Quoc V. Le
  8. Ofir Nachum

In Computational Theory and Mathematics

  1. Ofir Nachum
  2. Sergey Levine
  3. Lihong Li
  4. Bo Dai
  5. Shimon Whiteson
  6. Emma Brunskill
  7. Niao He
  8. Kavosh Asadi

In Reinforcement Learning

  1. Sergey Levine
  2. Ofir Nachum
  3. Lihong Li
  4. Shimon Whiteson
  5. Bo Dai
  6. Pieter Abbeel
  7. Nan Jiang
  8. Emma Brunskill

In Information Theory

  1. Bo Dai
  2. Ofir Nachum
  3. Joan Bas-Serrano
  4. Dale Schuurmans
  5. Lihong Li
  6. Sham M. Kakade
  7. Mohammad Norouzi
  8. Michael L. Littman

In Game & Decision Theory

  1. Craig Boutilier
  2. Bo Dai
  3. Ofir Nachum
  4. Shimon Whiteson
  5. Joan Bas-Serrano
  6. Mohammad Ghavamzadeh
  7. Dale Schuurmans
  8. Lihong Li

Citations in the corpus