An Alternative Softmax Operator for Reinforcement Learning

By Kavosh Asadi, Michael L. Littman. 2016

237 citations recorded by the source corpus.

Subfields: Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

1. Ofir Nachum
2. Sergey Levine
3. Lihong Li
4. Bo Dai
5. Dale Schuurmans
6. Nan Jiang
7. Aviral Kumar
8. George Tucker

In Optimization & Training

1. Niao He
2. Le Song
3. Bo Dai
4. Dale Schuurmans
5. Adam Gleave
6. Oliver Habryka
7. Ofir Nachum
8. Mohammad Norouzi

Citations in the corpus

• A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, methodology): “The research literature contains a wide variety of studies of operator alternatives to the Bellman operator, including the -greedy method [22], speedy Q-learning [3], policy iteration-like Q-learning [7], and the Boltzmann softmax operator and its variants [2].”
• Lipschitz Continuity in Model-based Reinforcement Learning (2018, background): “The following operators (Asadi & Littman, 2017) are Lipschitz with constants: Theorem 3.”
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning (2021, background): “While TICQ is not a contraction [5] (similar with the softmax operator), the Q-values are still within a reasonable range.”
• A Theory of Regularized Markov Decision Processes (2019, background): “P Its convex conjugate is Ω ∗ ( q s ) = ln a 1 |A| exp q s ( a ), that is the Mellowmax operator [Asadi and Littman, 2017].”
• Improving Policy Gradient by Exploring Under-appreciated Rewards (2016, background): “Some of these value-based methods have proposed a softening of Q-values which can be interpreted as adding a form of maximum-entropy regularizer (Asadi & Littman, 2016; Azar et al., 2012; Fox et al., 2016; Ziebart, 2010).”
• Recent advances in reinforcement learning in finance (2021, methodology): “For Boltzmann exploration [10, 105, 232], the agent draws actions from a Boltzmann distribution (softmax) over the learned Q value function, regulated by a temperature parameter.”
• Multi-task Maximum Entropy Inverse Reinforcement Learning (2018, methodology): “It models the demonstrations as being sampled from: a stochastic expert policy that is noisily optimal for: Note there can exist multiple solutions to these softmax Bellman equations (Asadi & Littman, 2017).”
• On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning (2017, background): “It is of interest to extend the results presented in this paper for generalized versions of the softmax function [60] or adopt a monotone operator theoretic approach to analyze alternative forms of choice maps [61].”
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation (2017, methodology): “First, we show T λ is also a contraction, as with the standard Bellman operator (Fox et al., 2016; Asadi & Littman, 2017): Proposition 1 (Contraction) T λ is a γ -contraction.”