Efficient softmax approximation for GPUs

By Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, Herv'e J'egou. 2016

297 citations recorded by the source corpus.

We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computational complexity. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Hardware and Architecture, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Efficiency & Compression

Citations in the corpus

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “We choose to use the same E for all word pieces because they are much more evenly distributed across documents compared to whole-word embedding, where having different embedding size (Grave et al. (2017); Baevski & Auli (2018); Dai et al. (2019) ) for different words is important.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “We adopted adaptive softmax and input representations (Baevski and Auli, 2018; Grave et al., 2016a).”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “We use an adaptive softmax for the output classi-ﬁer (Grave et al., 2017) for the word based models and regular softmax for the BPE based models.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “…de,fr } + numrefs.1+smooth.exp+test.wmt14/full+tok.13a+version.1.2.9 En–De En–Fr a. Gehring et al. (2017) 25.2 40.5 b. Vaswani et al. (2017) 28.4 41.0 c. Ahmed et al. (2017) 28.9 41.4 d. Shaw et al. (2018) 29 (Grave et al., 2017), and adaptive inputs (Baevski and Auli, 2019).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “Adaptive word representations are inspired by the adaptive softmax work Grave et al. (2017) which ﬁrst described a GPU friendly way to construct a hierarchical softmax and showed that it performs very competitively compared to a full softmax, while offering signiﬁcantly faster speed and a lower…”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “…across all its occurrences in the retrieved targets (items that do not appear in the retrieved targets have zero probability): Finally, we follow Grave et al. (2017a) and interpolate the nearest neighbor distribution p kNN with the model distribution p LM using a tuned parameter λ to produce…”
Adaptive Semiparametric Language Models (2021, methodology): “For word-based language modeling, we use adaptive softmax (Grave et al., 2017b).”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “[21] have shown that this strategy can improve modern language models like recurrent networks without retraining.”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “Since both datasets have large vocabularies, we use the adaptive softmax (Grave et al., 2016) for faster training.”
FastText.zip: Compressing text classification models (2016, methodology): “Linear classiﬁers often obtain state-of-the-art performance while being scalable (Agarwal et al., 2014; Joachims, 1998; Joulin et al., 2016; McCallum & Nigam, 1998).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “Training: To handle the large vocabulary of Wikitext-103, we follow Dauphin et al. (2017) and Baevski & Auli (2018) in using adaptive softmax (Grave et al., 2016) and adaptive input for computational efficiency.”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “For the Billion word benchmark we use an adaptive softmax output layer to reduce the computational burden of the large vocabulary (Grave et al., 2016; Press & Wolf, 2017) We train on 32 GPUs with batches of 65K tokens for 975K updates.”
Unsupervised Cross-lingual Representation Learning at Scale (2019, methodology): “With bigger models, we believe that using a vocabulary of up to 2 million tokens with an adaptive softmax (Grave et al., 2017; Baevski and Auli, 2018) should improve performance even further, but we leave this exploration to future work.”
Language Modeling with Gated Convolutional Networks (2016, methodology): “…et al., 2016) 30.6 32 GPUs BIG GLSTM-G4 (Kuchaiev & Ginsburg, 2017) 23.3 ∗ 8 GPUs LSTM-2048 (Grave et al., 2016a) 43.9 1 GPU 2-layer LSTM-2048 (Grave et al., 2016a) 39.8 1 GPU GCNN-13 38.1 1 GPU GCNN-14 Bottleneck 31.9 8 GPUs (Jozefowicz et al., 2016) which uses the full softmax, the adaptive…”
Learning to Write with Cooperative Discriminators (2018, methodology): “ADAPTIVELM Our first baseline is the same Adaptive Softmax (Grave et al., 2016) language model used as base generator in our framework (§3.”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, background): “Adaptive softmax (Joulin et al., 2017) uses the natural imbalance in word distributions (Zipf, 1949) to form clusters in a hierarchical model, which minimizes computation time.”
An Analysis of Neural Language Modeling at Multiple Scales (2018, methodology): “To address these issues, we use a modiﬁed version of the adaptive softmax (Grave et al., 2016b), extended to allow for tied weights (Inan et al., 2016; Press & Wolf, 2016).”