IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation

By Kishore Papineni, Salim Roukos, Tomás Ward, Wujie Zhu. 2001

292 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Furu Wei
2. Li Dong
3. Hangbo Bao
4. Nan Yang
5. Hsiao-Wuen Hon
6. Helen Wright Hastie
7. Jianfeng Gao
8. Miruna Clinciu

In Machine Translation

1. Li Dong
2. Jianfeng Gao
3. Furu Wei
4. Ming Zhou

Citations in the corpus

• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (2020, positive): “As shown in Table 5, we report BLEU (Papineni et al., 2002), METEOR(Banerjee & Lavie, 2005), and ROUGE (Lin, 2004) scores on two different data splits.”
• Beyond Data and Model Parallelism for Deep Neural Networks (2018, neutral): “…and a softmax layer WMT English-German (WMT) 19.67c 19.85c a top-1 accuracy for single crop on the validation dataset (higher is better). b word-level test perplexities on the Peen Treebank dataset (lower is better). c BLEU scores (Papineni et al., 2002) on the test dataset (higher is better).”
• Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals (2020, unknown): “We report BLEU scaled to 0 – 100 as is usual in most papers (although BLEU was originally de fi ned as 0 – 1 by Papineni et al. 28 ); the higher BLEU value, the better translation.”
• Adapted large language models can outperform medical experts in clinical text summarization (2023, unknown): “Given that BLEU measures sequence overlap, this result seems reasonable, as more text provides more ‘surface area’ for overlap; more text also reduces the brevity penalty that BLEU applies on generated sequences, which are shorter than the reference 42 .”
• A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (2021, methodology): “Furthermore, they do not use recall, rather a Brevity Penalty, which penalizes generated text for being “too short” (Papineni et al., 2001).”
• Linear Transformers Are Secretly Fast Weight Programmers (2021, unknown): “Table 1 shows the B LEU score (Papineni et al., 2002; Post, 2018) results.”
• Evaluating Large Language Models on Medical Evidence Summarization (2023, unknown): “In contrast, a low BLEU score implies that the generated summary is written differently from the reference summary.”
• Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction (2018, unknown): “Label smoothing helps to generate higher-scoring translations in terms of accuracy and BLEU score [48] for human languages, and also helps in terms of reaching higher top-1 accuracy in reaction prediction.”
• Sequence Level Training with Recurrent Neural Networks (2015, unknown): “One such metric is called BLEU (Papineni et al., 2002) for instance, which measures the n-gram overlap between the model generation and the reference text.”