Scalable Modified Kneser-Ney Language Model Estimation

By Kenneth Heafield, Ivan Pouzyrevsky, J. Clark, Philipp Koehn. 2013

632 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Generating Sentences by Editing Prototypes (2017, methodology): “KN5: a standard 5-gram Kneser-Ney language model in KenLM (Heafield et al., 2013).”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “For English, our language model is a Kneser-Ney smoothed 5-gram model with pruning that is trained using the KenLM toolkit [28] on cleaned text from the Common Crawl Repository3.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “We consider a 4-gram KenLM language model (Heaﬁeld et al., 2013) and a character based convolutional language model (Likhomanenko et al., 2019) and tune the models with the same protocol as Schneider et al. (2019).”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “53† 4-GRAM LM (Heafield et al., 2013) Baseline – – 3.”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, methodology): “We trained an n-gram language model using the KenLM toolkit (Heafield et al., 2013).”
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, methodology): “For comparing the speed, we report performance for sequence sizes as reported initially by Baidu, but also for longer sequence sizes, which corresponds to our average use 1http://www.torch.ch.”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, methodology): “Our baseline is an interpolated Kneser-Ney (KN) language model and we use the KenLM toolkit to train 5-gram models without pruning (Heaﬁeld et al., 2013).”
Latent Predictor Networks for Code Generation (2016, methodology): “for word alignments (Och and Ney, 2003), MERT for tuning (Sokolov and Yvon, 2011) and a 4-gram Kneser-Ney Smoothed language model (Heafield et al., 2013).”
The United Nations Parallel Corpus v1.0 (2016, methodology): “…n-grams pruned if they occur only once) and a 9-gram word-class language model with word-classes produced by word2vec (Mikolov et al., 2013) (3-grams and 4-grams are pruned if they occur only once, 5-grams and 6-grams if they occur only twice, etc.), both trained using KenLM (Heaﬁeld et al., 2013).”
Learning to Ask: Neural Question Generation for Reading Comprehension (2017, methodology): “We train a tri-gram language model on target side texts with KenLM (Heaﬁeld et al., 2013), and tune the system with MERT on dev set.”
Letter-Based Speech Recognition with Gated ConvNets (2017, methodology): “Our decoder relies on KenLM (Heaﬁeld et al., 2013) for the language modeling part.”
Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation (T) (2015, methodology): “We use following tools in this study: MeCab to tokenize Japanese sentences [29], pialign to train word alignments [22], KenLM to train Kneser-Ney smoothed language model [30], Moses to train and generate target sentences using the PBMT model [31], and Travatar to train and generate target sentences ”
Larger-Context Language Modelling with Recurrent Neural Network (2015, methodology): “Furthermore, we also report the result with the conventional, count-based n -gram language model with the modiﬁed Kneser-Ney smoothing with KenLM (Heaﬁeld et al., 2013 Each recurrent language model uses 1000 LSTM units and is trained with Adadelta (Zeiler, 2012) to maximize the log-likelihood…”