If MaxEnt RL is the Answer, What is the Question?
By Benjamin Eysenbach, Sergey Levine. 2019
66 citations recorded by the source corpus.
Experimentally, it has been observed that humans and animals often make decisions that do not maximize their expected utility, but rather choose outcomes randomly, with probability proportional to expected utility. Probability matching, as this strategy is called, is equivalent to maximum entropy reinforcement learning (MaxEnt RL). However, MaxEnt RL does not optimize expected utility. In this paper, we formally show that MaxEnt RL does optimally solve certain classes of control problems with variability in the reward function. In particular, we show (1) that MaxEnt RL can be used to solve a certain class of POMDPs, and (2) that MaxEnt RL is equivalent to a two-player game where an adversary chooses the reward function. These results suggest a deeper connection between MaxEnt RL, robust control, and POMDPs, and provide insight for the types of problems for which we might expect MaxEnt RL to produce effective solutions. Specifically, our results suggest that domains with uncertainty in the task goal may be especially well-suited for MaxEnt RL methods.
Subfields: Artificial Intelligence, Cognitive Neuroscience, Molecular Biology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Rodica Pop-Busui
- Jianyu Chen
- Ian Fox
- Masayoshi Tomizuka
- Shengbo Eben Li
- Joyce M. Lee
- Jenna Wiens
- Pravendra Singh
In Cognitive Neuroscience
In Molecular Biology
Citations in the corpus
- Interpretable End-to-End Urban Autonomous Driving With Latent Deep Reinforcement Learning (2020, methodology): “Now considering we are using a parametric function as the policy π φ , for example a deep neural network with weights φ , then the objective of MaxEnt RL can be written as : There are several reasons why we would like to use MaxEnt RL instead of standard RL [43].”
- Deep Reinforcement Learning for Closed-Loop Blood Glucose Control (2020, background): “Additionally, maximum entropy policies like the ones produced by SAC can do well in partially observed settings like our own (Eysenbach and Levine, 2019).”