A Structured Review of the Validity of BLEU

By Ehud Reiter. 2018

461 citations recorded by the source corpus.

The BLEU metric has been widely used in NLP for over 15 years to evaluate NLP systems, especially in machine translation and natural language generation. I present a structured review of the evidence on whether BLEU is a valid evaluation technique—in other words, whether BLEU scores correlate with real-world utility and user-satisfaction of NLP systems; this review covers 284 correlations reported in 34 papers. Overall, the evidence supports using BLEU for diagnostic evaluation of MT systems (which is what it was originally proposed for), but does not support using BLEU outside of MT, for evaluation of individual texts, or for scientific hypothesis testing.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

A Call for Clarity in Reporting BLEU Scores (2018, background): “Its weaknesses abound, and much has been written about them (cf. Callison-Burch et al. (2006); Reiter (2018)).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “As pointed out by Reiter and Belz (2009a), the evaluation meta-evaluations we draw on are most commonly conducted on summarization and machine translation (MT), but that there is an implicit”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “issue somewhat, these metrics still have a low correlation with human judgments (Reiter, 2018; Fabbri et al., 2020).”
Neural Generation for Czech: Data and Baselines (2019, methodology): “To obtain a deeper insight into the results and account for automatic metrics’ inaccuracy (Novikova et al., 2017; Reiter, 2018), we performed a detailed manual error analysis on a sample of 100 outputs produced by all systems except the ones with random baseline lexicalizers, which clearly perform p”
Survey of Hallucination in Natural Language Generation (2022, background): “Recently, various studies have illustrated that most conventional metrics used to measure the quality of writing are not adequate for quantifying the level of hallucination [156].”
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation (2019, background): “Over the years several studies have evaluated automatic metrics for measuring text generation performance (Callison-Burch et al., 2006; Stent et al., 2005; Belz and Reiter, 2006; Reiter, 2018; Liu et al., 2016; Kilickaya et al., 2017; Gatt and Krahmer, 2018).”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, neutral): “Finally, it must be noted that the results using automatic metrics are quite different from results obtained in human evaluation (see Section 8.4), which confirms previous findings (Novikova et al. 2017; Reiter 2018).”
Neural Text Summarization: A Critical Evaluation (2019, methodology): “…the authors put under scrutiny existing methodologies, datasets, or models were conducted by Callison-Burch et al. (2006, 2007); Tan et al. (2015); Post (2018) in machine translation, (Gkatzia and Mahamood, 2015; Reiter and Belz, 2009; Reiter, 2018) in natu-ral language generation, Lee et al. (2016”
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions (2020, methodology): “We selected papers for inclusion in this study following the PRISMA methodology (Moher et al., 2009) recently introduced to NLP by Reiter (2018) in his structured review of the validity of BLEU.”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “, 2002) and ROUGE (Lin, 2004) compute the scores based primarily on n-gram co-occurrence statistics, which are originally proposed for diagnostic evaluation of systems but not capable of evaluating text quality (Reiter, 2018), as they are not designed to measure if, and to what extent, the system an”
Evaluation of Text Generation: A Survey (2020, background): “Reiter (2018) report that there’s not good evidence to support that BLEU is the best metric for evaluating NLG systems other than machine translation.”
Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue (2019, methodology): “In order to overcome these challenges, we propose the use of structured meaning representations like those explored widely in (hybrid) rule-based NLG systems (Rambow et al., 2001; Reiter and Dale, 2000; Walker et al., 2007).”
Disembodied Machine Learning: On the Illusion of Objectivity in NLP (2021, background): “This exempliﬁes the overarching problem that there is rarely one single ‘gold’ label for a given document (Reiter 2018), yet most training and evaluation algorithms assume just that.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “…or a task-relevant part of the state vector as the goal, it is difﬁcult to deﬁne meaningful distance metrics in the space of language statements [7, 47, 55], and, consequently, difﬁcult to make the reward signal smooth by assigning partial credit (unlike, e.g., the (cid:96) p norm of the…”
Building Adaptive Acceptability Classifiers for Neural NLG (2021, background): “Moreover, such methods have not been found to correlate well with human evaluation of individual texts outside of the machine translation paradigm (Reiter, 2018).”
The ReproGen Shared Task on Reproducibility of Human Evaluations in NLG: Overview and Results (2021, background): “This is of particular concern in Natural Language Generation (NLG) where human evaluations have always played a central role (Reiter, 2018; Novikova et al., 2017).”
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation (2021, background): “model-generated text, which is critical for openended tasks given the unreliability of automatic metrics (Peng et al., 2017; Reiter, 2018; See et al., 2019), is frequently conducted on Amazon’s popular Mechanical Turk platform (AMT) to minimize cost and time.”
Syntax-Guided Controlled Generation of Paraphrases (2020, background): “…1983; Barzi-lay and Lee, 2003; Quirk et al., 2004; Hassan et al., 2007; Zhao et al., 2008; Madnani and Dorr, 2010; Wubben et al., 2010), they have recently been superseded by deep learning-based approaches (Prakash et al., 2016; Gupta et al., 2018; Li et al., 2019, 2018; Kumar et al., 2019).”
Best practices for the human evaluation of automatically generated text (2019, background): “1 In light of this criticism, it has been argued that automated metrics are not suitable to assess linguistic properties (Scott and Moore, 2007), and Reiter (2018) discouraged the use of automatic metrics as a (primary) evaluation metric.”
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020, background): “BLEU (Papineni et al., 2002a) has remained mostly unchanged since it was proposed in 2002, but its correlation with human scores has changed each year over ten years of evaluation (2006 to 2016) on the English–German and German–English language pairs at WMT (Reiter, 2018).”