A Call for Clarity in Reporting BLEU Scores

By Matt Post. 2018

3,550 citations recorded by the source corpus.

The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to “the” BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SACREBLEU, to facilitate this.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, neutral): “0 [Post, 2018] with “exp” smoothing and “intl” tokenization.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “In fact, even within the same language, tokenization discrepancies pose significant challenges to reliable evaluation (Post, 2018).”
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (2018, background): “(Post, 2018) reported that subtle differences in preprocessing schemes can widely change BLEU scores.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2006) and detokenized BLEU with SacreBLEU1 (Post, 2018).”
Scaling Neural Machine Translation (2018, methodology): “We measure case-sensitive tokenized BLEU with multi-bleu.pl 2 and de-tokenized BLEU with SacreBLEU 3 (Post, 2018).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “produces a unique hash, inspired by the SacreBLEU framework (Post, 2018).”
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (2021, methodology): “Prior research has attempted to generate effective prompts using separate models [19].”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “For most languages, we use the moses tokenizer (Koehn et al., 2007).6 For Chinese we use the SacreBLEU tokenizer (tok zh) and convert all traditional characters generated by the model to simplified characters using HanziConv7 (Post, 2018),8 for Indian languages 2. https://github.com/facebookresearch”
Complete Multilingual Neural Machine Translation (2020, positive): “All BLEU (Papineni et al., 2002) scores are calculated with sacreBLEU (Post, 2018).”
Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information (2020, neutral): “We also report de-tokenized BLEU with SacreBLEU (Post, 2018) for a fair comparison with previous works.”
Reformer: The Efficient Transformer (2020, methodology): “We additionally report detokenized BLEU scores as computed by sacreBLEU (Post, 2018).”
Toward Gender-Inclusive Coreference Resolution (2019, methodology): “To quantify this bias, we use the sacreBLEU toolkit [Post 2018] and compute Bleu scores between the ground truth reference outputs and those same references where all she pronouns were replaced with morphologically equivalent he forms (none of these datasets contain neopronouns and analysis of a…”
Tagged Back-Translation (2019, methodology): “We therefore report strictly SacreB LEU 1 , using the reference implementation from Post (2018), which aims to standardize B LEU evaluation.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We evaluate using the sacrebleu (Post, 2018) implementation of BLEU (Papineni et al.”
Root Mean Square Layer Normalization (2019, methodology): “We evaluate translation quality with case-sensitive detokenized BLEU score reported by sacrebleu [21] 4 .”
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019, methodology): “(Papineni et al., 2002) for all language pairs, except from EN to ZH, where we use SacreBLEU (Post, 2018).”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “, 2002), SacreBLEU (Post, 2018), METEOR (Banerjee & Lavie, 2005); (2) model-based semantic metrics such as BertScore (Zhang et al.”
WT5?! Training Text-to-Text Models to Explain their Predictions (2020, neutral): “0 (Post, 2018) with exp smoothing and intl tokenization.”