Bleu: a Method for Automatic Evaluation of Machine Translation
By Kishore Papineni, Salim Roukos, Tomás Ward, Wei-Jing Zhu. 2002
33,928 citations recorded by the source corpus.
Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Graham Neubig
- Yejin Choi
- Yoshua Bengio
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yejin Choi
- Kyunghyun Cho
- Percy Liang
- Noah A. Smith
In Evaluation & Benchmarks
- Mohit Bansal
- Jianfeng Gao
- Yejin Choi
- Elizabeth Clark
- Graham Neubig
- Luke S. Zettlemoyer
- Lucia Specia
- Percy Liang
Citations in the corpus
- Multi-task Sequence to Sequence Learning (2015, methodology): “Additionally, we also compute test BLEU scores (Papineni et al., 2002) for the translation task.”
- Effective Approaches to Attention-based Neural Machine Translation (2015, background): “Translation performances are reported in case-sensitive BLEU (Papineni et al., 2002) on newstest2014 (2737 sentences) and newstest2015 (2169 sentences).”
- Sequence to Sequence Learning with Neural Networks (2014, methodology): “We used the cased BLEU score [24] to evaluate the quality of our translations.”
- MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “BLEU score (Papineni et al., 2002) is used extensively in machine translation where one can compare the quality of candidate translations from the reference.”
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books (2015, methodology): “Note that BLEU does not contribute significantly to the performance and is of optional use.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “For all translation tasks, we report the BLEU score [Papineni et al., 2002] as provided by SacreBLEU v1.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “We follow standard practice and report tokenized case-sensitive BLEU (Papineni et al., 2002) with compound splitting (Vaswani et al.”
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, neutral): “The final results are reported in BLEU (Papineni et al., 2002) with language-specific settings, see appendix A.”
- Natural Questions: A Benchmark for Question Answering Research (2019, background): “Free-form text answers allow more flexibility in providing abstractive answers, but lead to difficulties in evaluation (BLEU score [Papineni et al., 2002] is used).”
- Massively Multilingual Neural Machine Translation (2019, background): “These different setups enable us to examine the effect of the number of translation tasks on the translation quality as measured in BLEU (Papineni et al., 2002).”
- Model Cards for Model Reporting (2018, background): “translation system could compare BLEU scores [40] across demographic groups, and a model card for a speech recognition system could compare word-error rates.”
- The Natural Language Decathlon: Multitask Learning as Question Answering (2018, methodology): “We evaluate with a corpus-level BLEU score [Papineni et al., 2002] on the 2013 and 2014 test sets as validation and test sets, respectively.”
- A Call for Clarity in Reporting BLEU Scores (2018, methodology): “In machine translation research, the predictions are made by models whose development is the focus of the research, and the measurement, more often than not, is done via BLEU (Papineni et al., 2002).”
- RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “Instead of evaluating the accuracy, researchers need to use F1 score, BLEU (Papineni et al., 2002) or ROUGE (Lin and Hovy, 2003) as metrics, which measure the overlap between the prediction and ground truth answers since the questions come without candidate spans.”
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “N-gram based metrics (Papineni et al., 2002; Dod-dington, 2002; Wang et al., 2016; Popovi´c, 2015) that measure lexical overlap require tokenization which is highly affected by language specific factors such as alphabet size and morphological complexity.”
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, methodology): “We used the case-sensitive BLEU score (Papineni et al., 2002) as an evaluation metric.”
- Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (2018, methodology): “We used the case sensitive BLEU score (Papineni et al., 2002) as an evaluation metric.”
- NewsQA: A Machine Comprehension Dataset (2016, methodology): “BLEU is a precision-based metric popular in machine translation that uses a weighted average of variable length phrase matches ( n -grams) against the reference sentence (Papineni et al., 2002).”
- Unsupervised Pretraining for Sequence to Sequence Learning (2016, methodology): “Evaluation on the validation set was with case-sensitive BLEU (Papineni et al., 2002) on tokenized text using multi-bleu.”