IRSTLM: an open source toolkit for handling large scale language models

By Marcello Federico, Nicola Bertoldi, Mauro Cettolo. 2008

369 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, background): “In the context of text, several works have shown that compression does not signiﬁcantly reduce the performance of models [17, 24, 30].”
Colorless Green Recurrent Networks Dream Hierarchically (2018, methodology): “…target forms (singular or plural); second, a 5-gram model with Kneser-Ney smoothing ( KN , Kneser and Ney, 1995) trained using the IRSTLM package (Federico et al., 2008) and queried us-ing KenLM (Heaﬁeld, 2011); and third, a 5-gram LSTM , which only had access to windows of ﬁve tokens (Chelba…”
The Kaldi Speech Recognition Toolkit (2011, methodology): “In our recipes, we have used the IRSTLM toolkit [20] for purposes like LM pruning.”
WIT3: Web Inventory of Transcribed and Translated Talks (2012, methodology): “…lexicalized reordering models were trained on parallel training data; taking into account the limited amount of training data, 4-gram LMs smoothed through the improved Kneser-Ney technique (Chen and Good-man, 1999) were estimated on monolingual texts via the IRSTLM toolkit (Federico et al., 2008).”
UGENT-LT3 SCATE System for Machine Translation Quality Estimation (2015, methodology): “For the PoS LM, we used IRSTLM with Witten-Bell smoothing (Federico et al., 2008) as the modified Kneser-Ney smoothing, which is used by KENLM, is not well defined when there are no singletons (Chen and Goodman 1999), which leads to modeling issues in the PoS corpus.”
PROMT Translation Systems for WMT 2016 Translation Tasks (2016, methodology): “The IRSTLM toolkit (Federico et al., 2008) is used to build language models, which are scored using KenLM (Heaﬁeld, 2011) in the decoding process.”
Scalable Modified Kneser-Ney Language Model Estimation (2013, methodology): “Compared with IRSTLM, our toolkit used 16.4% of the CPU time, 9.0% of the wall time, and 16.6% of the RAM.”
N-gram Counts and Language Models from the Common Crawl (2014, methodology): “We emphasize that this approach is not new, but rather standard practice recommended by IRSTLM (Federico et al., 2008).”
Domain Adaptation for Statistical Machine Translation with Monolingual Resources (2009, methodology): “A 5-gram language model was trained on the target side of the training parallel corpus using the IRSTLM toolkit (Federico et al., 2008), exploiting Modiﬁed Kneser-Ney smoothing, and quantizing both probabilities and backoff weights.”
KenLM: Faster and Smaller Language Model Queries (2011, methodology): “The size of TRIE is particularly sensitive to dlog2 c1e, so vocabulary filtering is quite effective at reducing model size.”
Postech’s System Description for Medical Text Translation Task (2014, methodology): “For each monolingual corpus, we used a five-gram language model, which was built by IRSTLM toolkit 2 (Federico, Bertoldi, & Cettolo, 2008) with improved Kneser Ney smoothing (Chen & Goodman, 1996; Kneser & Ney, 1995).”
UGENT-LT3 SCATE Submission for WMT16 Shared Task on Quality Estimation (2016, methodology): “For building the PoS LM, we used TreeTagger (Schmid, 1995) to obtain the PoS tags on the target (DE) data and IRSTLM (Federico et al., 2008) for building the LM.”
Modelling pronominal anaphora in statistical machine translation (2010, methodology): “It is a phrase-based SMT system based on the Moses decoder with phrase tables trained on the Europarl version 5 and news-commentary10 parallel corpora and a 6gram language model trained on the monolingual News corpus provided by the workshop organisers with the IRSTLM language modelling toolkit [15]”
Translating Video Content to Natural Language Descriptions (2013, methodology): “To estimate the fluency of the descriptions we use IRSTLM [6] which is based on n-gram statistics of TACoS.”
Experiments in Medical Translation Shared Task at WMT 2014 (2014, methodology): “The language model was created with open source IRSTLM toolkit (Federico et al., 2008) using all the English in-domain data (monolingual and parallel).”
The IWSLT 2015 Evaluation Campaign (2015, methodology): “2 Translation and lexicalized reordering models were trained on the parallel training data by means of the Moses toolkit; 5-gram LMs with improved Kneser-Ney smoothing were estimated on the target side of the training data with the IRSTLM toolkit [33].”