A unified view of entropy-regularized Markov decision processes
By Gergely Neu, Anders Jonsson, Vicenç Gómez. 2017
318 citations recorded by the source corpus.
We propose a general framework for entropy-regularized average-reward reinforcement learning in Markov decision processes (MDPs). Our approach is based on extending the linear-programming formulation of policy optimization in MDPs to accommodate convex regularization functions. Our key result is showing that using the conditional entropy of the joint state-action distributions as regularization yields a dual optimization problem closely resembling the Bellman optimality equations. This result enables us to formalize a number of state-of-the-art entropy-regularized reinforcement learning algorithms as approximate variants of Mirror Descent or Dual Averaging, and thus to argue about the convergence properties of these methods. In particular, we show that the exact version of the TRPO algorithm of Schulman et al. (2015) actually converges to the optimal policy, while the entropy-regularized policy gradient methods of Mnih et al. (2016) may fail to converge to a fixed point. Finally, we illustrate empirically the effects of using various regularization techniques on learning performance in a simple reinforcement learning setup.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning, Information Theory, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Quoc V. Le
- Ofir Nachum
In Computational Theory and Mathematics
In Reinforcement Learning
In Information Theory
- Bo Dai
- Ofir Nachum
- Joan Bas-Serrano
- Dale Schuurmans
- Lihong Li
- Sham M. Kakade
- Mohammad Norouzi
- Michael L. Littman
In Game & Decision Theory
- Craig Boutilier
- Bo Dai
- Ofir Nachum
- Shimon Whiteson
- Joan Bas-Serrano
- Mohammad Ghavamzadeh
- Dale Schuurmans
- Lihong Li
Citations in the corpus
- Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “Alternatively, one may use the techniques in Section 6, writing the policy optimization problem analogous to (67) as max One may solve this problem via its Lagrangian or by adding an appropriate regularizer and applying Fenchel-Rockafellar duality (Neu et al., 2017).”
- AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “The regularization we employ is inspired by previous uses of regularization in RL. Adding regularization to MDPs (Neu et al., 2017; Geist et al., 2019) has been investigated for many different purposes in the literature, including exploration (de Farias and Van Roy, 2000; Haarnoja et al., 2017,…”
- Boosting the Actor with Dual Critic (2017, background): “The duality view has also been exploited in Neu et al. (2017).”
- Efficient Exploration Through Bayesian Deep Q-Networks (2018, background): “In practical policy gradient, even though the optimal policy, given Markovian assumption, needs to be deterministic, the policy regularization is a dominant approach to make the policy stochastic and preserve the exploration thorough the stochasticity of the policy [51], [52].”
- Understanding the impact of entropy on policy optimization (2018, methodology): “Perhaps some work should be directed on alternate smoothing techniques: Santurkar et al. (2018) suggests that techniques like batch normalization also smooth the objective function and might be able to replicate some of the performance benefits.”
- SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, methodology): “While Eqns (4) and (5) are inspired by Nestorov smoothing technique, they can also be derived from other principles (Rawlik et al., 2012; Fox et al., 2016; Neu et al., 2017; Nachum et al., 2017; Asadi & Littman, 2017).”
- Expected Policy Gradients for Reinforcement Learning (2018, background): “This follows because adding entropy regularisation to the objective of optimising the total discounted reward in an RL setting corresponds to shifting the reward function by a term proportional to log π ( a | s ) (Neu et al., 2017; Nachum et al., 2017).”
- Cold-Start Reinforcement Learning with Softmax Policy Gradient (2017, background): “(7) is the dual of the entropy-regularized policy search (REPS) objective [5, 16] L(q) = Eq[R] + KL(q|pθ).”
- Provably Efficient Maximum Entropy Exploration (2018, background): “Although the above reduction is outlined for an exhaustive policy class, similar reductions are possible for linearlyparameterized policy classes (Peters et al., 2010; Neu et al., 2017).”
- Faster saddle-point optimization for solving large-scale Markov decision processes (2019, methodology): “, 1999; Konda and Tsitsiklis, 1999) and various “entropy-regularized” learning algorithms (Peters et al., 2010; Zimin and Neu, 2013; Neu et al., 2017).”
- CAQL: Continuous Action Q-Learning (2019, background): “However, the sub-optimality gap of this soft policy scales with the interval and dimensionality of the action space (Neu et al., 2017).”
- A Lyapunov-based Approach to Safe Reinforcement Learning (2018, background): “Finally, instead of exactly solving the LP problem for policy optimization in (6), one may approximate this solution by solving its entropy regularized counterpart [28].”
- Lipschitz Continuity in Model-based Reinforcement Learning (2018, background): “1 was proven by Littman & Szepesv´ari (1996), and 2 is proven several times (Fox et al., 2016; Asadi & Littman, 2017; Nachum et al., 2017; Neu et al., 2017).”
- Revisiting Design Choices in Proximal Policy Optimization (2020, background): “This objective has not been studied empirically but has been the main focus of theoretical studies around PPO [11, 14, 16].”