Strategies for Training Large Vocabulary Neural Language Models

By David Grangier, Michael Auli. 2015

143 citations recorded by the source corpus.

Training neural network language models over large vocabularies is still computationally very costly compared to count-based models such as Kneser-Ney. At the same time, neural language models are gaining popularity for many applications such as speech recognition and machine translation whose success depends on scalability. We present a systematic comparison of strategies to represent and train large vocabularies, including softmax, hierarchical softmax, target sampling, noise contrastive estimation and self normalization. We further extend self normalization to be a proper estimator of likelihood and introduce an efficient variant of softmax. We evaluate each method on three popular benchmarks, examining performance on rare words, the speed/accuracy trade-off and complementarity to Kneser-Ney.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Adaptive Input Representations for Neural Language Modeling (2018, background): “Word-based models are particularly challenging since computing probabilities for all 800K words of the BILLION WORD benchmark is still a substantial part of the overall computation (Chen et al., 2016).”
FastText.zip: Compressing text classification models (2016, background): “One idea is to condition the size of the vectors (both for the input features and the labels) based on their frequency (Chen et al., 2015; Grave et al., 2016).”
Language Modeling with Gated Convolutional Networks (2016, methodology): “We evaluated on the Gigaword dataset following Chen et al. (2016) to compare with fully connected models.”
Efficient softmax approximation for GPUs (2016, methodology): “Our method is compared to: (1) the full soft-max, (2) the hierarchical softmax with frequency binning (HSM freq) and similarity-based binning (HSM sim), (3) importance sampling (Bengio et al., 2003b; Bengio & Sen´ecal, 2008) and (4) the differentiated softmax (Chen et al., 2015).”
Exploring Sparsity in Recurrent Neural Networks (2017, neutral): “And in language modeling the size of the non-embedding parameters (mostly in the recurrent layers) have exploded even as various ways of hand engineering sparsity into the embeddings have been explored in J´ozefowicz et al. (2016) and Chen et al. (2015a).”
Pre-training Tasks for Embedding-based Large-scale Retrieval (2020, methodology): “Sampled Softmax has been widely used in language modeling (Chen et al., 2016; Grave et al., 2017), recommendation systems (Yu et al., 2017; Krichene et al., 2019) and extreme classiﬁcation (Blanc & Rendle, 2018; Reddi et al., 2019).”
Fast Parametric Learning with Activation Memorization (2018, background): “Prior literature on the softmax operator for language modelling computational efficiency (Chen et al., 2015; Grave et al., 2016a) or tricks such as smoothing across many softmax layers (Yang et al.”
Generalizing and Hybridizing Count-based and Neural Language Models (2016, background): “…other hand, these neural LMs also come at the cost of increased computational complexity at both training and test time, and even the largest reported neural LMs (Chen et al., 2015; Williams et al., 2015) are trained on a fraction of the data of their count-based counterparts (Brants et al., 2007).”
Online normalizer calculation for softmax (2018, background): “Differentiated Softmax [4] and SVD-Softmax [5] replace the projection layer - which is usually just a matrix multiplication - with more computationally efficient alternatives.”
Recurrent Memory Networks for Language Modeling (2016, unknown): “Chen et al. (2015) show that n-gram model outperforms a popular feed-forward language model (Bengio et al., 2003) on a one billion word benchmark (Chelba et al., 2013).”
Neural information retrieval: at the end of the early years (2017, background): “The first group of remedies such as Hierarchical Softmax [147] and differentiated softmax [36] propose updates to the softmax layer architectures for efficient computation.”
Recurrent Memory Network for Language Modeling (2016, unknown): “Chen et al. (2015) show that n-gram model outperforms a popular feed-forward language model (Bengio et al., 2003) on a one billion word benchmark (Chelba et al., 2013).”
Neural Machine Translation via Binary Code Prediction (2017, methodology): “As a result, we can control the actual computation for the hybrid model to be much smaller than the standard softmax complexity O(HV ), The idea of separated prediction of frequent words and rare words comes from the differentiated softmax (Chen et al., 2016) approach.”