Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

By Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W. Cohen. 2017

428 citations recorded by the source corpus.

We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Model Architectures

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Inspired by previous work [38], given a sequence x = [x1, · · · , xT ], we define a set of target-context pairs of interest, I = {(x,U)}, where U is a set of tokens in x that form a context of x.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “, 2016a) , and enriching the output distribution family (Yang et al., 2017).”
Learning Representations by Maximizing Mutual Information Across Views (2019, methodology): “One weakness of this approach is that it limits the rank of the set of belief shifts our model can represent when the vector space is ﬁnite-dimensional, as was previously addressed in the context of language modeling by introducing mixtures [Yang et al., 2018].”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, background): “(Yang et al., 2018b) propose a similar architecture for language modeling.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, background): “Mixture of Softmaxes (MoS) (Yang et al., 2017) improves the expressiveness of a single softmax operation by instead computing a linear combination over softmaxes, each weighted by learned coeﬃcients.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “…LSTM (Melis et al., 2017) Hyper-parameters Search 24 59.5 LSTM (Yang et al., 2018) VD, WT, ℓ 2 , AWD, MoC 22 57.6 LSTM (Merity et al., 2017) VD, WT, ℓ 2 , AWD 24 57.3 LSTM (Yang et al., 2018) VD nodes and switch the non-linearity into identity or sigmoid, the perplexity increases up to 8 points.”
DARTS: Differentiable Architecture Search (2018, background): “9 24 – manual LSTM + 15 softmax experts (Yang et al., 2018) 66.”
Neural Architecture Optimization (2018, methodology): “3 / LSTM + averaged weight drop + Mixture of Softmax + weight penalty + weight tying [47] 22M 56.”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “We also evaluated our method within the MoS framework by directly following the experimental settings in Yang et al. (2018), except we replace the original softmax function with our adversarial softmax function.”
Evaluating the Search Phase of Neural Architecture Search (2019, methodology): “Furthermore, when training an architecture from scratch, we follow [33, 23] and first make the use of standard SGD to speed up training, and then change to average SGD to improve convergence.”
Random Search and Reproducibility for Neural Architecture Search (2019, methodology): “[48] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W.”
FRAGE: Frequency-Agnostic Word Representation (2018, methodology): “We do experiments on two widely used datasets [25, 26, 41], Penn Treebank (PTB) [27] and WikiText-2 (WT2) [26].”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “We compare AWD-LSTM-DOC with AWD-LSTM (Merity et al., 2018) and AWD-LSTM-MoS (Yang et al., 2018).”
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, background): “…years now, neural language models (NLMs) (Graves, 2013; Jozefowicz et al., 2016; Grave et al., 2017a; Dauphin et al., 2017; Melis et al., 2018; Yang et al., 2018) have consistently outperformed classical n -gram models, an improvement often attributed to their ability to model long-range…”
Dynamic Evaluation of Neural Sequence Models (2017, methodology): “Since publishing our code, the state-ofthe-art on PTB and Wikitext-2 has been further improved by applying our dynamic evaluation implementation on top of a stronger base model (Yang et al., 2018).”
Relational recurrent neural networks (2018, background): “RNNs, and most notably LSTMs, have proven to be state-of-the-art on many competitive language modeling benchmarks such as Penn Treebank [27, 28], WikiText-103 [29, 30], and the One Billion Word Benchmark [31, 32].”
An Analysis of Neural Language Modeling at Multiple Scales (2018, methodology): “, 2018) and mixture-of-softmaxes (Yang et al., 2018) but since our goal is to evaluate the underlying model, we employ no such strategies in addition.”
Insertion Transformer: Flexible Sequence Generation via Insertion Operations (2019, neutral): “We posit that the language modeling softmax bottleneck identiﬁed by Yang et al. (2018) poses even greater challenges for our setup.”