Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain

By Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu (羅致翹) Lo, Craig Alan Stewart, George F. Foster, Alon Lavie, Ondrej Bojar. 2021

206 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Similarly, Freitag et al. (2021a) demonstrate that non-expert annotations often lead to mistaken claims of super-human model performance, when expert annotators correctly identify issues in the generated texts.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “One of the findings from last years shared task is that metrics struggle to accurately penalize translations with errors in reversing negation (Freitag et al., 2021b).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “…(SOTA) MT systems; (ii) we use a SOTA automatic metric (BLEURT) instead of BLEU which has been demonstrated to be suboptimal for high-quality translations (Kocmi et al., 2021; Freitag et al., 2021b); and (iii) we conduct an expert-based human evaluation with detailed ar X iv :2 21 1.”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, unknown): “Machine Translation Task As WMT20’s standard practice (Mathur et al., 2020), we compute the correlations of each evaluation metric to the segment-and system-level human scores, on WMT20 and WMT21, with MQM-based labels (Freitag et al., 2021a).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “• Additional Training Data : We encouraged the participants to make use of existing MQM annotations for newstest2020 (Freitag et al., 2021a) 3 , and the MQM annotations from the WMT21 Metrics Task (Freitag et al., 2021b) to improve and/or test their metrics.”
Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, methodology): “We design the main prompts based on the DA+SQM template used for human assessment of translation quality as implemented in the Appraise framework (Federmann, 2018) for WMT22 (Kocmi et al., 2022), building on previous work conducted by Freitag et al. (2021a).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “One common practice is to apply in-batch token insertions or replacements to the pivot sentence (Fu et al., 2022).”