Sigsoftmax: Reanalysis of the Softmax Bottleneck
By Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, Shuichi Adachi. 2018
78 citations recorded by the source corpus.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an output activation function for breaking the softmax bottleneck without additional parameters. We re-analyze the softmax bottleneck from the perspective of the output set of log-softmax and identify the cause of the softmax bottleneck. On the basis of this analysis, we propose sigsoftmax, which is composed of a multiplication of an exponential function and sigmoid function. Sigsoftmax can break the softmax bottleneck. The experiments on language modeling demonstrate that sigsoftmax and mixture of sigsoftmax outperform softmax and mixture of softmax, respectively.
Subfields: Artificial Intelligence, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Qiang Liu
- Luke S. Zettlemoyer
- Yoshua Bengio
- Percy Liang
In Model Architectures
- Ruslan Salakhutdinov
- Quoc V. Le
- Zihang Dai
- Zhilin Yang
- Qiang Liu
- Yiming Yang
- Chengyue Gong
- William W. Cohen
Citations in the corpus
- Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et al., 2018).”
- Improving Neural Language Modeling via Adversarial Training (2019, methodology): “We use the processed version provided by Mikolov et al. (2010) that is widely used for this dataset (e.g., Mer-ity et al., 2018a; Yang et al., 2018; Kanai et al., 2018; Gong et al., 2019).”
- Transformer-XL: Language Modeling with Longer-Term Dependency (2018, background): “…al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et al., 2018).”
- Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, background): “We also find that the proposed mixture model can alleviate the softmax bottleneck problem in natural language which has been revealed by [46, 20].”