Open Source Toolkit for Statistical Machine Translation: Factored Translation Models and Lattice Decoding

By Philipp Koehn, Marcello Federico, Wade Shen, Nicola Bertoldi, Chris Callison-Burch, Ondrej Bojar, B. Cowan, Chris Dyer, Hieu T. Hoang, Richard Zens and 3 more. 2006

733 citations recorded by the source corpus.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Learned in Translation: Contextualized Word Vectors (2017, methodology): “Each is tokenized using the Moses Toolkit [Koehn et al., 2007].”
Cross-lingual Language Model Pretraining (2019, methodology): “For all other languages, we use the tokenizer provided by Moses (Koehn et al., 2007), falling back on the default English tokenizer when necessary.”
Deep Learning Scaling is Predictable, Empirically (2017, methodology): “We clean and tokenize the data set using Moses (Koehn et al. (2007)) as described by Luong et al. (2017).”
Pointer Sentinel Mixture Models (2016, methodology): “Normalization and tokenization were performed using the Moses tokenizer (Koehn et al., 2007), slightly augmented to further split numbers (8,600 → 8 @,@ 600) and with some additional minor fixes.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “For most languages, we use the moses tokenizer (Koehn et al., 2007).6 For Chinese we use the SacreBLEU tokenizer (tok zh) and convert all traditional characters generated by the model to simplified characters using HanziConv7 (Post, 2018),8 for Indian languages 2.…”
Large Memory Layers with Product Keys (2019, methodology): “We tokenized the data using the tokenizer provided by the Moses toolkit [25].”
Moses: Open Source Toolkit for Statistical Machine Translation (2007, background): “Initial results show the potential benefit of factors for statistical machine translation, (Koehn et al. 2006) and (Koehn and Hoang 2007).”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “A smoothed variant, SENTBLEU (Koehn et al., 2007) is computed at the sentence level.”
Addressing the Rare Word Problem in Neural Machine Translation (2014, methodology): “For more than a decade, the standard MT approach [14] has been subject to intensive research and resulted in better systems over time [13, 4, 3, 8].”
Are Sixteen Heads Really Better than One? (2019, methodology): “2018, we compute BLEU scores on the tokenized output of the model using Moses (Koehn et al., 2007).”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “The text is tokenized and processed using the Moses tokenizer (Koehn et al., 2007), frequently used for machine translation, and features a vocabulary of over 30,000 words.”
Movie Description (2016, methodology): “In the second step it translates the SR to a sentence using Statistical Machine Translation (SMT) (Koehn et al. 2007).”
Latent Predictor Networks for Code Generation (2016, methodology): “We follow the same approach and create a phrase-based (Koehn et al., 2007) model and a hierarchical model (or PCFG) (Chiang, 2007) as benchmarks for the work presented here.”
On Layer Normalization in the Transformer Architecture (2020, methodology): “We ﬁrst segment documents into sentences with Spacy 3 ; Then, we normalize, lower-case, and tokenize texts using Moses (Koehn et al., 2007) and apply BPE(Sennrich et al., 2016).”
Unsupervised Translation of Programming Languages (2020, methodology): “For instance, Nguyen et al. [36] trained a Phrase-Based Statistical Machine Translation (PBSMT) model, Moses [27], on a Java-C# parallel corpus.”
TGIF: A New Dataset and Benchmark on Animated GIF Description (2016, neutral): “[ 15] to learn the SMT model.”