Understanding the impact of entropy on policy optimization

By Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, Dale Schuurmans. 2018

317 citations recorded by the source corpus.

Entropy regularization is commonly used to improve policy optimization in reinforcement learning. It is believed to help with \emph{exploration} by encouraging the selection of more stochastic policies. In this work, we analyze this claim using new visualizations of the optimization landscape based on randomly perturbing the loss function. We first show that even with access to the exact gradient, policy optimization is difficult due to the geometry of the objective function. Then, we qualitatively show that in some environments, a policy with higher entropy can make the optimization landscape smoother, thereby connecting local optima and enabling the use of larger learning rates. This paper presents new tools for understanding the optimization landscape, shows that policy entropy serves as a regularizer, and highlights the challenge of designing general-purpose policy optimization algorithms.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

Citations in the corpus

DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “This also increases the smoothness of the loss landscape, which is crucial for performance of RL algorithms (Ahmed et al., 2019).”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “, 2017; Williams & Peng, 1991), where causal entropy is added to the rewards, making the Q-function smoother and facilitating optimization (Ahmed et al., 2018).”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “Someworks have revealed that the objective with entropy regularization enjoys the smoother optimization landscape and the faster convergence rate, which builds connections between RL with probabilistic graphical models and convex optimization [1].”
Batch Reinforcement Learning Through Continuation Method (2021, methodology): “As pointed out in [3], even with exact gradients, the objective function V ⇡ ( ⇢ ) is still difﬁcult to optimize due to its highly non-smooth landscape.”
On the Global Convergence Rates of Softmax Policy Gradient Methods (2020, background): “(iii) Empirical results suggest that entropy helps optimization (Ahmed et al., 2019).”
The problem with DDPG: understanding failures in deterministic environments with sparse rewards (2019, methodology): “Using new tools such as the ones provided in Ahmed et al. (2019), recent analyses of the deadly triad such as Achiam et al. (2019) as well as simple, easily visualized benchmarks and our own tools, for future work we aim to conduct deeper and more exhaustive analysis of all the instability factors…”
A Closer Look at Deep Policy Gradients (2018, background): “McCandlish et al. (2018) investigates gradient noise in large-batch settings, and Ahmed et al. (2018) investigates the role of entropy regularization (which we do not study) on optimization.”
Reinforcement Learning: Theory and Algorithms (2019, background): “, 2016] introduces the entropy regularizer (also see [Ahmed et al., 2019] for a more detailed empirical investigation).”
Neural Architecture Search for Class-incremental Learning (2019, background): “To obtain better reward signals for learning, the rewards are normalized into [-1,1] range within all architectures sampled at time t. Lastly, we add an entropy term to the reward function in order to improve policy optimization (Ahmed et al. 2019).”