On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
By Bolin Gao, Lacra Pavel. 2017
384 citations recorded by the source corpus.
In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the Lipschitz and co-coercivity properties of the softmax function. We then demonstrate the usefulness of these properties through an application in game-theoretic reinforcement learning.
Subfields: Artificial Intelligence, Public Health, Environmental and Occupational Health, Sociology and Political Science, Reinforcement Learning, Game & Decision Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Stefano Ermon
- Yoshua Bengio
- Mohit Bansal
- Quoc V. Le
- Percy Liang
- Ruslan Salakhutdinov
- Jianfeng Gao
In Public Health, Environmental and Occupational Health
- Michael L. Littman
- Karsten M. Borgwardt
- Franccois Fleuret
- Apoorv Vyas
- Dipendra Kumar Misra
- Angelos Katharopoulos
- Kavosh Asadi
In Sociology and Political Science
- Michael L. Littman
- Junjie Yan
- Aditya Grover
- Franccois Fleuret
- Bernard Ghanem
- Apoorv Vyas
- Dipendra Kumar Misra
- Sepp Hochreiter
In Reinforcement Learning
- Sergey Levine
- Kavosh Asadi
- Dipendra Kumar Misra
- J. Urgen Schmidhuber
- Aditya Grover
- Stefano Ermon
- Michael L. Littman
- Pieter Abbeel
In Game & Decision Theory
- Stefano Ermon
- Aditya Grover
- Michael Widrich
- Michael L. Littman
- Eric Wang
- Franccois Fleuret
- Bernard Ghanem
- Apoorv Vyas
Citations in the corpus
- Regularisation of neural networks by enforcing Lipschitz continuity (2018, background): “The Lipschitz constant of the softmax is one (Gao and Pavel, 2017).”
- Hopfield Networks is All You Need (2020, methodology): “From a physical point of view, the lse function represents the “free energy” in statistical thermodynamics [29].”
- cosFormer: Rethinking Softmax in Attention (2022, background): “…key properties that affect its empirical performance: (i) elements in the attention matrix are non-negative (Tsai et al., 2019; Katharopoulos et al., 2020); (ii) the non-linear re-weighting scheme acts as a stabilizer for the attention weights (Titsias, 2016; Gao & Pavel, 2017; Jang et al., 2016).”
- DeeperGCN: All You Need to Train Deeper GCNs (2020, background): “Energy-Based Learning [LeCun et al., 2006], Knowledge Distillation [Hinton et al., 2015] and Rein-forcement Learning [Gao and Pavel, 2017].”
- AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, background): “where Ls is the Lipschitz constant of the normalization function [58], [59].”
- Stochastic Optimization of Sorting Networks via Continuous Relaxations (2019, background): “As shown in Gao & Pavel (2017), the softmax function may be equivalently defined as soft max( z/τ ) = arg max x ∈ ∆ n − 1 (cid:104) x, z (cid:105) − τ =1 x i log x i .”
- Structure-Aware Transformer for Graph Representation Learning (2022, background): “The first inequality comes from a simple matrix norm inequality, and the second inequality uses the fact that softmax function is 1/ √ doutLipschitz (see e.g. Gao & Pavel (2017)).”
- Fast Transformers with Clustered Attention (2020, positive): “Given that softmax (·) has Lipschitz constant less than 1 [10], ∥∥softmax (QiKT )− softmax (QjKT )∥∥2 ≤ ∥∥QiKT −QjKT∥∥2 ≤ ‖K‖2 (8)”
- Lipschitz Continuity in Model-based Reinforcement Learning (2018, positive): “(cid:80) Gao & Pavel (2017) showed that ρ is Lipschitz: Using their result, we can further show: + (cid:107) x 2 (cid:107) 2 ρ ( x 1 ) − ρ ( x 2 ) 2 (Cauchy-Shwartz) ≤ ρ ( x 1 ) 2 x 1 − x 2 (cid:107) 2 + (cid:107) x 2 (cid:107) 2 β x 1 − x 2 (cid:107) 2 from Eqn 7) dividing both sides by x 1 − x 2…”