A New Softmax Operator for Reinforcement Learning

By Kavosh Asadi, Michael L. Littman. 2016

10 citations recorded by the source corpus.

A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a single maximum utility decision. The Boltzmann softmax operator is the most commonly used softmax operator in this setting, but we show that this operator is prone to misbehavior. In this work, we study an alternative softmax operator that, among other properties, is both a non-expansion (ensuring convergent behavior in learning and planning) and differentiable (making it possible to improve decisions via gradient descent methods). We provide proofs of these properties and present empirical comparisons between various softmax operators.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ofir Nachum
2. Bo Dai
3. Sergey Levine
4. Lihong Li
5. Dale Schuurmans
6. Mohammad Norouzi
7. Pieter Abbeel
8. Le Song

In Management Science and Operations Research

1. Ofir Nachum
2. Bo Dai
3. Lihong Li
4. Dale Schuurmans
5. Mohammad Norouzi
6. Niao He
7. Gergely Neu
8. Lin Xiao

In Reinforcement Learning

1. Ofir Nachum
2. Sergey Levine
3. Bo Dai
4. Lihong Li
5. Dale Schuurmans
6. Mohammad Norouzi
7. George Tucker
8. Niao He

In Optimization & Training

1. Niao He
2. Le Song
3. Gergely Neu
4. Bo Dai
5. Vicenç Gómez
6. Anders Jonsson
7. Dale Schuurmans
8. Mohammad Norouzi

Citations in the corpus

• Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “More recently Fox et al. (2016) use a log-weighted-average-exp and Asadi & Littman (2016) propose a mellowmax operator, defined as log-averageexp.”
• Smoothed Dual Embedding Control (2017, methodology): “ty, Haarnoja et al. (2017) reformulates the update as an optimization which is approximated by samples from stein variational gradient descent (SVGD) sampler. Another related algorithm is proposed in Asadi and Littman (2016) the intractability issue of the log-sum-exp operator, named as ‘mellowmax’,”
• A unified view of entropy-regularized Markov decision processes (2017, methodology): “We note that analogous results have been proven several times in the literature, see, e.g., (Fox et al., 2016; Asadi and Littman, 2016).”
• Improving Policy Gradient by Exploring Under-appreciated Rewards (2016, background): “Some of these value-based methods have proposed a softening of Q-values which can be interpreted as adding a form of maximum-entropy regularizer (Asadi & Littman, 2016; Azar et al., 2012; Fox et al., 2016; Ziebart, 2010).”