BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies

By Shihao Ji, S. V. N. Vishwanathan, Nadathur Rajagopalan Satish, Michael J. Anderson, Pradeep K. Dubey. 2015

77 citations recorded by the source corpus.

We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a new sampling strategy which significantly reduces computation while improving stability, sample efficiency, and rate of convergence. One way to understand BlackOut is to view it as an extension of the DropOut strategy to the output layer, wherein we use a discriminative training loss and a weighted sampling scheme. We also establish close connections between BlackOut, importance sampling, and noise contrastive estimation (NCE). Our experiments, on the recently released one billion word language modeling benchmark, demonstrate scalability and accuracy of BlackOut; we outperform the state-of-the art, and achieve the lowest perplexity scores on this dataset. Moreover, unlike other established methods which typically require GPUs or CPU clusters, we show that a carefully implemented version of BlackOut requires only 1-10 days on a single machine to train a RNNLM with a million word vocabulary and billions of parameters on one billion words. Although we describe BlackOut in the context of RNNLM training, it can be used to any networks with large softmax output layers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Language Modeling with Gated Convolutional Networks (2016, methodology): “Test PPL Hardware Sigmoid-RNN-2048 (Ji et al., 2015) 68.3 1 CPU Interpolated KN 5-Gram (Chelba et al., 2013) 67.6 100 CPUs Sparse Non-Negative Matrix LM (Shazeer et al., 2014) 52.9 - RNN-1024 + MaxEnt 9 Gram Features (Chelba et al., 2013) 51.3 24 GPUs LSTM-2048-512 (Jozefowicz et al., 2016) 43.7 32 ”
Efficient softmax approximation for GPUs (2016, methodology): “4 RNN-2048 + BlackOut sampling [20] 68.”
Recurrent Neural Networks with Top-k Gains for Session-based Recommendations (2017, neutral): “ the case of RNNs for session-based recommendations. Another work related to sampling large output spaces in deep networks for efficient loss computations for language models is the ’blackout’ method [11], where essentially a sampling procedure similar to the one used in [7] is applied in order to e”
Online normalizer calculation for softmax (2018, methodology): “Sampled-based approximations, such as Importance Sampling [9], Noise Contrastive Estimation [10], and Blackout [11] accelerate training by running Softmax on select elements of the original vector.”
An Introduction to Neural Information Retrieval (2018, unknown): “We refer readers to [Ji et al., 2015] for more rigorous discussions on the relationship between IS, NCE, and DropOut.”
Conditional Noise-Contrastive Estimation of Unnormalised Models (2018, unknown): “…e.g. uniform or Gaussian distributions, have successfully been used (Gutmann & Hyv¨arinen, 2012; Mnih & Teh, 2012), the estimation performance of NCE depends on the distribution chosen and more tailored distributions were found to typically yield better results, see e.g. (Ji et al., 2016).”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, background): “Neural language models often work best in combination with traditional N -gram models (Mikolov et al., 2011; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2016; Shazeer et al., 2015), since the former excel at generalization while the latter ensure memorization.”
A Deep Learning-based Radar and Camera Sensor Fusion Architecture for Object Detection (2019, methodology): “gle neurons, we simultaneously deactivate all input neurons for the camera image data, for random training steps. This is done at a rate of 0:2 of all training images. We call this technique BlackIn. [35] introduced BlackOut which is inspired by dropout on the ﬁnal layer of the network. The absence ”
Neural Machine Translation with Source-Side Latent Graph Parsing (2017, methodology): “For the BlackOut objective (Ji et al., 2016), the number of the negative samples was set to 2,000 for the small and medium training datasets, and 2,500 for the large training dataset.”
Variable length word encodings for neural translation models (2016, methodology): “The parameters of the LSTM LMs are configured close to what is described in Jozefowicz et al. (2016); we set the learning rate at 0.”
Tree-to-Sequence Attentional Neural Machine Translation (2016, methodology): “It would take about 11 days without using the BlackOut sampling.”
One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities (2016, methodology): “Regarding related work, there exist several other methods that try to deal with the high cost of softmax such as methods that attempt to perform the exact computations [9, 19], methods that change the model based on hierarchical or stick-breaking constructions [16, 13] and sampling-based methods [1,”
Complementary Sum Sampling for Likelihood Approximation in Large Scale Classification (2017, methodology): “Alternative, non-likelihood based training approaches such as Noise Contrastive Estimation [7, 14], Negative Sampling [12] and BlackOut [10] have been recently introduced.”
Learning to Parse and Translate Improves Neural Machine Translation (2017, methodology): “To reduce computational overhead, we use Black-Out (Ji et al., 2015) with 2000 negative samples and α = 0 .”