KenLM: Faster and Smaller Language Model Queries

By Kenneth Heafield. 2011

1,599 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, methodology): “…sentence length of 80, grow-diag-ﬁnal-and symmetrization of GIZA++ alignments, an interpolated Kneser-Ney smoothed 5-gram language model with KenLM (Heaﬁeld, 2011) used at runtime, hierarchical lexicalized re-ordering (Galley and Manning, 2008), a lexically-driven 5-gram operation sequence model…”
Findings of the 2016 Conference on Machine Translation (2016, methodology): “For language modeling we used the KenLM toolkit (Heafield, 2011) for standard n-gram modeling with an n-gram length of 5.”
Neural Generation for Czech: Data and Baselines (2019, methodology): “(6)We used the implementation in the KenLM toolkit (Heafield, 2011).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “We use a 5 -gram Kneser-Ney model as implemented in the KenLM library (Heaﬁeld, 2011) because of its efﬁciency to process large quantity of data.”
Unbounded cache model for online language modeling with open vocabulary (2017, background): “In the context of text, several works have shown that compression does not signiﬁcantly reduce the performance of models [17, 24, 30].”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, methodology): “A 4-gram language model is trained with the resulting corpus using the KenLM toolkit (Heafield, 2011) and the top 200k words as vocabulary.”
Parallel Corpus Filtering via Pre-trained Language Models (2020, methodology): “Denkowski et al. (2012), Dyer et al. (2010) and Heaﬁeld (2011) use language models and word alignments to determine how likely sentences are to be a good translation of another.”
Unsupervised Speech Recognition (2021, methodology): “The WFST is composed with a 4-gram language model (Heafield, 2011) pruned to keep only 4-grams occurring more than 3 times.”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “We decode each ﬁne-tuned model using the word-level n -gram language model (LM) from the target domain built with KenLM [41] and report its WER.”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “We build n-gram language models for decoding (when specified) using KenLM (Heafield, 2011).”
MLS: A Large-Scale Multilingual Dataset for Speech Research (2020, methodology): “We have trained 3-gram and 5-gram language models (LM) for all the languages in our dataset using the KenLM toolkit [18].”
Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, methodology): “We train an n-gram LM using KenLM toolkit [31] and a Transformer LM similar to [26] for each dataset independently using their in-domain LM training corpus.”
Iterative Pseudo-Labeling for Speech Recognition (2020, neutral): “As an ablation study, we train 5-gram LMs on GB \ LV \ LS and GB \ LV with top 200k words in each and without pruning using the KenLM toolkit [28].”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, positive): “We decode with a python wrapped version of the wav2letter++ decoder [18], using a 4-gram KenLM [22] language model trained on the unaligned text set.”
Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “We provide a thin wrapper on top of KenLM for n-gram language models [14].”
Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “Language Modeling: Both in the source and target domains we learn smoothed n-gram language models using KenLM (Heaﬁeld, 2011), although neural models could also be considered.”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “(2020), we trained SentencePiece unigram tokenizers (Kudo, 2018) followed by KenLM 5-gram models after tokenization (Heafield, 2011) on Wikipedia article openings for every language that was extracted from OSCAR.”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “We consider three baselines: ﬁrst, a unigram baseline, which picks the most frequent form in the training corpus out of the two candidate target forms (singular or plural); second, a 5-gram model with Kneser-Ney smoothing ( KN , Kneser and Ney, 1995) trained using the IRSTLM package (Federico et al.”
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task (2017, methodology): “We estimated trigram models using KenLM (Heafield, 2011) from two different datasets: (1) the entire training stories, and (2) only the endings from the training stories.”