Results of the WMT20 Metrics Shared Task

By Nitika Mathur, Johnny Tian-Zheng Wei, Markus Freitag, Qingsong Ma, Ondrej Bojar. 2020

170 citations recorded by the source corpus.

This paper presents the results of the WMT20 Metrics Shared Task. Participants were asked to score the outputs of the translation systems competing in the WMT20 News Translation Task with automatic metrics. Ten research groups submitted 27 metrics, four of which are reference-less “metrics”. In addition, we computed five baseline metrics, including sentBLEU, BLEU, TER and using the SacreBLEU scorer. All metrics were evaluated on how well they correlate at the system-, document- and segment-level with the WMT20 official human scores. We present an extensive analysis on influence of different reference translations on metric reliability, how well automatic metrics score human translations, and we also flag major discrepancies between metric and human scores when evaluating MT systems. Finally, we investigate whether we can use automatic metrics to flag incorrect human ratings.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “However, as metrics and systems improved further, the DA annotations proved insufficient to identify a “best” metric (Mathur et al., 2020), which led to another major change to the methodology (Freitag et al., 2021b).”
Learning Compact Metrics for MT (2021, neutral): “E XTENDED , three SOTA metrics from WMT Metrics ’20 (Mathur et al., 2020b).”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, background): “Each year, the WMT Metrics shared task seeks to determine the automatic metric that better correlates with human evaluations (Mathur et al., 2020).”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, methodology): “We use the Williams significance test (Williams, 1959; Graham and Baldwin, 2014) following the practice adopted by WMT (Mathur et al., 2020) to identify correlation differences are statistically significant.”
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021, background): “This indicates that r-BLEU correlates with human judgments better at the corpus-level than at the sentence-level, as in machine translation evaluation (Mathur et al., 2020).”
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (2021, methodology): “Comparing the pretrained methods, ESIM is the best performing method in general (Mathur et al., 2020b), while COMET is the best performing method when removing the suspicious system.”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, background): “…(Bawden et al., 2020b) • chat translation (Farajian et al., 2020) • lifelong learning (Barrault et al., 2020) 1http://www.statmt.org/wmt20/ • metrics (Mathur et al., 2020) • parallel corpus filtering (Koehn et al., 2020) • quality estimation (Specia et al., 2020a) • robustness (Specia et…”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “Task Works/Challenges that release human scores MT WMT-20 [86] and all the previous year WMTs AS Fabbri et al.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We use Kendall’s tau-like correlation, typically used for DA RR (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020b), for evaluating the primary submissions on the challenge sets.”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “The WMT shared metrics competition (Ma et al., 2018, 2019; Mathur et al., 2020) is the main submission entry for evaluation metrics.”
The statistical advantage of automatic NLG metrics at the system level (2021, methodology): “To recount a perplexing anecdote, in the most recent edition of the WMT metrics shared task (Mathur et al., 2020b), initial human evaluation disagreed with most metrics on a pairwise prediction of two translation systems.”
Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics (2022, methodology): “To this end, in this work, we firstly re-evaluate ten popular metrics on many datasets from News domain between 2010 and 2019, with the Error Number evaluation method (Mathur et al., 2020a).”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “…such as B LEU (Papineni et al., 2002) have been adopted by the MT community as a primary form of MT evaluation yet, recently, these classic metrics have been outperformed by metrics based on large pretrained models such as BERT (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020).”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, background): “…0.12 0.07 0.11 0.12 0.1 0.13 0.24 0.15 0.04 0.03 0.07 0.04 0.06 0.03 0.03 0.01 0.1 0.08 0.12 0.09 0.05 0.03 0.07 0.14 0.06 0.04 0.02 0.21 0.22 0.24 0.01 0.08 0.01 0.13 0.21 0.2 0.21 0.25 0.2 0.17 0.22 0.15 For MT, we refer to the results of the WMT 2020 shared metrics task (Mathur et al., 2020b).”
Dialect-robust Evaluation of Generated Text (2022, methodology): “Metrics We calculate the segment-level agreement with human ratings and report DaRR (Mathur et al., 2020), a robust variant of Kendall Tau.”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, unknown): “Machine Translation Task As WMT20’s standard practice (Mathur et al., 2020), we compute the correlations of each evaluation metric to the segment-and system-level human scores, on WMT20 and WMT21, with MQM-based labels (Freitag et al., 2021a).”
Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, background): “Metrics shared tasks Mathur et al. (2020) and Fre-itag et al. (2021b) changed this back to the 2011 version.”