Strategies for training large scale neural network language models

By Tomas Mikolov, Anoop Deoras, Daniel Povey, Lukáš Burget, Jan Honza Cernocký. 2011

567 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Similar to [17], to compare different model architectures we define first the computational complexity of a model as the number of parameters that need to be accessed to fully train the model.”
Recurrent Neural Network Regularization (2014, background): “Language modeling is the ﬁrst task where RNNs have achieved substantial success (Mikolov et al., 2010; 2011; Pascanu et al., 2013).”
From captions to visual concepts and back (2014, methodology): “The ME features are implemented in a hash table as in [27].”
Efficient softmax approximation for GPUs (2016, methodology): “Similar to the frequency binning hierarchical softmax (Mikolov et al., 2011c), this information can be exploited to reduce the computation cost.”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “Inorder to further reduce the perplexity, we combine the RNN model’s output with the output from a Maximum Entropy language model [26], simultaneously learned from the training corpus.”
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks (2015, background): “However, these same architectures have been shown to be unstable to small, well sought, perturbations of the images.”
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition (2017, methodology): “The maximum entropy RNN (ME-RNN) model [19] has direct connections between the input and output layers of an RNN layer.”
Deep Learning: Methods and Applications (2014, methodology): “Empirical comparisons with other state-ofthe-art counting-based N -gram LMs show much better performance of RNNLM in the perplexity measure, as reported in [247, 248] and [245].”
Sequence-to-sequence neural net models for grapheme-to-phoneme conversion (2015, background): “The use of a side-conditioned language model [11] is attractive for its simplicity, and apparent performance, and these successes complement other recent work in which neural networks have advanced the state-of-the-art, for example in language modeling [12, 13], language understanding [14], and pars”
Chinese Poetry Generation with Recurrent Neural Networks (2014, methodology): “To speed up training, we employed word-classing (Mikolov et al., 2011b).”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, background): “Consequently, the training times for some of the largest models reported in literature are of the order of weeks (Mikolov et al., 2011; Williams et al., 2015).”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “More recent work on neural network LM with deep architectures can be found in (Le et al., 2010, 2011a; Mikolov et al., 2010; Mikolov et al., 2011; Mikolov, 2012).”