Empirical Evaluation and Combination of Advanced Language Modeling Techniques

By Tomas Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, Jan Honza Cernocký. 2011

337 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “…scale LM has shown that RNNs are great in combination with N-grams, as they may have different strengths that complement N-gram models, but worse when considered in isolation (Mikolov et al., 2011; Mikolov, 2012; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2015a; Shazeer et al., 2015).”
Theano: new features and speed improvements (2012, methodology): “In Figure 2, we present a benchmark of a simple recurrent network, on Theano and RNNLM5, a C++ implementation of recurrent networks for language modeling.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, background): “Perplexity is the exponentiation of the average cross entropy of a corpus (Mikolov et al., 2011).”
Efficient Estimation of Word Representations in Vector Space (2013, background): “For example, neural network based language models significantly outperform N-gram models [1, 25, 16].”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “In particular, smoothed N-gram models [32, 34] have been the dominant type of models historically, achieving good performance in practice [44].”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “When θ is equal to zero, the probability distribution over the history is uniform, and our model is equivalent to a unigram cache model (Kuhn & De Mori, 1990).”
Representation Learning: A Review and New Perspectives (2012, background): “The neural net language model was also improved by adding recurrence to the hidden layers (Mikolov et al., 2011), allowing it to beat the state-of-the-art (smoothed n-gram models) not only in terms of perplexity (exponential of the average negative log-likelihood of predicting the right next word,…”
Character-Aware Neural Language Models (2015, background): “While NLMs have been shown to outperform count-based n-gram language models (Mikolov et al. 2011), they are blind to subword information (e.”
Efficient softmax approximation for GPUs (2016, methodology): “Similar to the frequency binning hierarchical softmax (Mikolov et al., 2011c), this information can be exploited to reduce the computation cost.”
A fast and simple algorithm for training neural probabilistic language models (2012, background): “In the last few years neural language models have become competitive with n -grams and now routinely out-perform them (Mikolov et al., 2011).”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, background): “Although neural LMs have begun to rival or even surpass traditional n -gram LMs (Mnih and Hin-ton, 2009; Mikolov et al., 2011), they have not yet been widely adopted in large-vocabulary applications such as MT, because standard maximum likelihood estimation (MLE) requires repeated summa-tions over…”
Big Neural Networks Waste Capacity (2013, background): “enge 2010 contains 1000 object categories and 1.2 million training images. 1Krizhevsky et al. (2012) reduced by almost one half the error rate on the 1000-class ImageNet object recognition benchmark 2Mikolov et al. (2011) reduced perplexity on WSJ by 40% and speech recognition absolute word error ra”
Context dependent recurrent neural network language model (2012, methodology): “Extensive comparison of performance of advanced language modeling techniques on this setup is given in [3].”
Recent Advances in Recurrent Neural Networks (2017, methodology): “In this approach, words are assigned to specific categories based on their unigram frequency and only words belonging to the predicted class are evaluated in the output layer [86].”
SUBWORD LANGUAGE MODELING WITH NEURAL NETWORKS (2011, methodology): “Alternat ively, we can apply neural network language models [8] (NNLM) to this p roblem, because NNLMs have been reported to achieve the state-o f-th art performance on several standard word-level language mo delling tasks [9, 10, 11].”
Automated Crowdturfing Attacks and Defenses in Online Review Systems (2017, background): “In contrast, a generative RNN model can generate much more coherent text [2, 43], but still falls short for larger types of content [50].”