Re-evaluating the Role of Bleu in Machine Translation Research

By Chris Callison-Burch, Miles Osborne, Philipp Koehn. 2006

487 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, neutral): “Some text generation tasks fail to meet this criteria due to issues with automatic metrics like ROUGE and BLEU (Callison-Burch et al., 2006; Liu et al., 2016, i.a.).”
A Call for Clarity in Reporting BLEU Scores (2018, neutral): “Its weaknesses abound, and much has been written about them (cf. Callison-Burch et al. (2006); Reiter (2018)).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, positive): “Since BLEU was introduced, dozens of papers have shown that automatic metrics have poor correlations with human judgments of quality (in addition to those cited above, see, e.g., Callison-Burch et al. (2006)).”
Learning Compact Metrics for MT (2021, neutral): “Since these metrics focus solely on surface-level aspects of the generated text, they correlate poorly with human evaluation, especially when models are producing high-quality text (Belz and Reiter, 2006; Callison-Burch et al., 2006; Ma et al., 2019; Mathur et al., 2020a).”
Towards a Better Metric for Evaluating Question Generation Systems (2018, positive): “Similar studies (Callison-Burch et al., 2006; Liu et al., 2016) have already shown that these metrics do not correlate well with fluency, adequacy, coherence but in this work, we focus on answerability.”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, neutral): “Designing effective automatic evaluation metrics that are highly correlated with human judgment remains a difﬁcult challenge [7], [42], [45], [46].”
CIDEr: Consensus-based image description evaluation (2014, neutral): “ness has been repeatedly questioned [22, 11, 4, 18].”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, neutral): “In particular, we investigate two approaches: word based similarity metrics and word-embedding based similarity metrics.”
Why We Need New Evaluation Metrics for NLG (2017, neutral): “…by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al., 2016), machine translation (MT) (Callison-Burch et al., 2006), and image captioning (Elliott and Keller, 2014; Kilickaya et al., 2017).”
(Meta-) Evaluation of Machine Translation (2007, neutral): “However, recent work suggests that Bleu’s correlation with human judgments may not be as strong as previously thought (Callison-Burch et al., 2006).”
Language Models for Image Captioning: The Quirks and What Works (2015, neutral): “We also note that the METEOR scores are close for all four models, and that it is well-documented in several NLP communities that BLEU scores do not always correlate with human quality judgments (Callison-Burch et al., 2006; Reiter and Belz, 2009; Hodosh et al., 2013).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, neutral): “studies in these fields have suggested that the correspondence is somewhat weak (e.g., Dorr et al., 2004; Callison-Burch et al., 2006; Caporaso et al., 2008).”
Manual and Automatic Evaluation of Machine Translation between European Languages (2006, neutral): “We confirm the finding by Callison-Burch et al. (2006) that the rule-based system of Systran is not adequately appreciated by BLEU.”
code2vec: learning distributed representations of code (2018, neutral): “While there are alternative metrics in the literature, such as accuracy and BLEU score, they are problematic because accuracy counts even mostly-correct predictions as completely incorrect, and BLEU score tends to favor short predictions, which are usually uninformative [Callison-Burch et al. 2006].”
Visual question answering: Datasets, algorithms, and future challenges (2016, neutral): “ evaluation. Each of these metrics has limitations. BLEU, the most widely used metric, is known to have the same score for large variations in sentence structure with largely varying semantic content [25]. For captions generated in [26], BLEU scores ranked machine generated captions above human capt”
Identifying Well-formed Natural Language Questions (2018, neutral): “However, its worth noting that an increase in well-formedness doesn’t guarantee an improved BLEU score, as the oracle sentence maximizing the BLEU score might be fairly non-wellformed (Callison-Burch et al., 2006).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, neutral): “The ultimate goal of automatic machine learning model evaluation is usually stated as capturing human judgements (Callison-Burch et al., 2006; Hill et al., 2015; Vedantam et al., 2015; Liu et al., 2016).”
Evaluation of Text Generation: A Survey (2020, neutral): “It has been argued that although BLEU has significant advantages, it may not be the ultimate measure for improved machine translation quality (Callison-burch & Osborne, 2006).”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, neutral): “However, in terms of performance, there exists a large gap between the evaluation of the metrics and human judgment of evaluation [20, 30, 74].”