Moses: Open Source Toolkit for Statistical Machine Translation
By Philipp Koehn, Hieu T. Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, B. Cowan, Wade Shen, C. Moran, Richard Zens and 4 more. 2007
6,539 citations recorded by the source corpus.
We describe an open-source toolkit for statistical machine translation whose novel contributions are (a) support for linguistically motivated factors, (b) confusion network decoding, and (c) efficient data formats for translation models and language models. In addition to the SMT decoder, the toolkit also includes a wide variety of tools for training, tuning and applying the system to many translation tasks.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Kyunghyun Cho
- Noah A. Smith
- Yoshua Bengio
In Natural Language Processing
- Graham Neubig
- Mohit Bansal
- Luke S. Zettlemoyer
- Kyunghyun Cho
- Jianfeng Gao
- Yonatan Belinkov
- Christopher D. Manning
- Philipp Koehn
Citations in the corpus
- Learned in Translation: Contextualized Word Vectors (2017, methodology): “Each is tokenized using the Moses Toolkit [Koehn et al., 2007].”
- Cross-lingual Language Model Pretraining (2019, methodology): “For all other languages, we use the tokenizer provided by Moses [24], falling back on the default English tokenizer when necessary.”
- Deep Learning Scaling is Predictable, Empirically (2017, methodology): “We clean and tokenize the data set using Moses (Koehn et al. (2007)) as described by Luong et al. (2017).”
- Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, methodology): “In order to create systems for the medical translation tasks, we used phrase-based Moses with exactly the same settings as for the news translation task, including the OSM (Durrani et al., 2011), and compound splitting Koehn and Knight (2003) for German source.”
- SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, methodology): “We to-kenize each of the datasets with the MOSES tokenizer (Koehn et al., 2007) and convert all files to UTF-8 encoding.”
- Findings of the 2015 Workshop on Statistical Machine Translation (2015, positive): “The stories were translated by the professional translation agency Capita, funded by the EU Framework Programme 7 project MosesCore, and by Yandex, a Russian search engine company.”
- Pointer Sentinel Mixture Models (2016, methodology): “Normalization and tokenization were performed using the Moses to-kenizer (Koehn et al., 2007), slightly augmented to further split numbers (8,600 → 8 @,@ 600) and with some additional minor fixes.”
- Beyond English-Centric Multilingual Machine Translation (2020, methodology): “For most languages, we use the moses tokenizer (Koehn et al., 2007).6 For Chinese we use the SacreBLEU tokenizer (tok zh) and convert all traditional characters generated by the model to simplified characters using HanziConv7 (Post, 2018),8 for Indian languages 2.…”
- Large Memory Layers with Product Keys (2019, methodology): “t observe any overfitting, and increasing the 5 model capacity systematically led to a better performance on the validation set. We tokenized the data using the tokenizer provided by the Moses toolkit [26]. To reduce the vocabulary size, we use fastBPE5 to apply Byte Pair Encoding (BPE) [39], with 60”
- Generating Sentences by Editing Prototypes (2017, background): “tence similarity and sentence-level analogies. 1 Introduction The ability to generate sentences is a core component of many NLP systems, including machine translation (Kalchbrenner and Blunsom, 2013; Koehn et al., 2007), summarization (Hahn and Mani, 2000; Nallapati et al., 2016), speech recognition”
- A Convolutional Encoder Model for Neural Machine Translation (2016, methodology): “1) and include a phrase-based system (Koehn et al., 2007).”
- Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “For significance tests we used approximate randomization (Clark et al., 2011), for BLEU score evaluation (lowercased) the multi-bleu script of the Moses decoder (Koehn et al., 2007), for TER computation the tercom tool (Snover et al., 2006).”