BLEURT: Learning Robust Metrics for Text Generation

By Thibault Sellam, Dipanjan Das, Ankur P. Parikh. 2020

1,948 citations recorded by the source corpus.

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgment. We propose BLEURT, a learned evaluation metric for English based on BERT. BLEURT can model human judgment with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG data set. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “We compare GPT-judge to metrics that make use of ROUGE1 [26] or BLEURT [37].”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Taking the example from Figure 2, we could state that “When the model summarizes news articles from the same source, but with COVID-19 related content, we expect quality drops of 20±5% according to BLEURT.”
Learning Compact Metrics for MT (2021, positive): “E XTENDED (Sellam et al., 2020a), based on XLM-RoBERTa (Conneau and Lample, 2019; Conneau et al., 2020a) and mBERT (Devlin et al., 2019) respectively.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “Filters can be applied along axes (see BLEURT axis in Figure 2) and the filtered selection is highlighted through bold lines.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “, 2020), and BLEURT (Sellam et al., 2020), but also with detailed human evaluations to really understand the challenges that still must be overcome.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “…“human-likeness” or “engagingness”, either by automatically comparing with a human ground-truth reference, e.g., by using similarity metrics such as BLEURT (Sellam et al., 2020) or BERTscore (Zhang et al., 2020a), or by asking humans to evaluate this manually (Deriu et al., 2020; Li et al., 2019).”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “We use the official evaluation script13 and report BLEU, METEOR, TER, MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020b) and BLEURT (Sellam et al., 2020).”
Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, methodology): “8Our experiments show that other metrics, e.g. BLEURT (Sellam et al., 2020) are also correlated with ROUGE-L, which has also been used in generative QA tasks. that incorporate different elements of the instructions(§4.2).9 Note that BART is fine-tuned with the Tseen tasks, while GPT3 uses no…”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “BLEURT (Sellam et al., 2020) is a fully model-based approach combining large-scale synthetic pretraining and domain specific finetuning.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “…compromise: models of human preference like pairwise learned preference models (Ouyang et al., 2022), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) have significantly improved correlation with human judgment compared to earlier metrics (BLEU, METEOR, etc.), and are cheap to…”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “Furthermore, we also use MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020a), and BLEURT (Sellam et al., 2020), which are new metrics that incorporate semantics rather than surface forms using contextual embeddings.”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) and PARENT (Dhingra et al., 2019”
Scaling Laws for Neural Machine Translation (2021, methodology): “To ensure that this observation truly reﬂects the generation quality of the models (as opposed to potential biases of BLEU score), we repeat our analysis with BLEURT score (Sellam et al., 2020a;b).”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “For instance, example 5 of built on learned models (Wiseman et al., 2017; Ma et al., 2019; Sellam et al., 2020) for appropriate evaluation.”
PaLM 2 Technical Report (2023, methodology): “, 2020): We use BLEURT9 (Sellam et al., 2020) as a SOTA automatic metric instead of BLEU (Papineni et al.”
Hurdles to Progress in Long-form Question Answering (2021, neutral): “Other possible ﬁxes include learning task-speciﬁc metrics to measure semantic overlap (Sellam et al., 2020) or metrics to check factual correctness (Zhang et al., 2020) and faithfulness to input (Wang et al., 2020; Durmus et al., 2020; Zhou et al., 2020).”