Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges

By Qingsong Ma, Johnny Tian-Zheng Wei, Ondrej Bojar, Yvette Graham. 2019

191 citations recorded by the source corpus.

This paper presents the results of the WMT19 Metrics Shared Task. Participants were asked to score the outputs of the translations systems competing in the WMT19 News Translation Task with automatic metrics. 13 research groups submitted 24 metrics, 10 of which are reference-less “metrics” and constitute submissions to the joint task with WMT19 Quality Estimation Task, “QE as a Metric”. In addition, we computed 11 baseline metrics, with 8 commonly applied baselines (BLEU, SentBLEU, NIST, WER, PER, TER, CDER, and chrF) and 3 reimplementations (chrF+, sacreBLEU-BLEU, and sacreBLEU-chrF). Metrics were evaluated on the system level, how well a given metric correlates with the WMT19 official manual ranking, and segment level, how well the metric correlates with human judgements of segment quality. This year, we use direct assessment (DA) as our only form of manual evaluation.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “The 2019 shared task introduced a quality estimation task in accordance with the DA data collection technique, illustrating how the human evaluation techniques can influence the design of metrics (Ma et al., 2019).”
Learning Compact Metrics for MT (2021, positive): “1 Following established approaches (Ma et al., 2018, 2019), we utilize the human ratings from the previous years’ shared tasks for training.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “The WMT shared metrics competition (Bojar et al., 2016; Ma et al., 2018, 2019) has inspired 5 Do those results imply that BLEU and ROUGE should be removed from future versions of B LEURT ?”
ToTTo: A Controlled Table-To-Text Generation Dataset (2020, background): “For instance, example 5 of built on learned models (Wiseman et al., 2017; Ma et al., 2019; Sellam et al., 2020) for appropriate evaluation.”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, background): “…in separate papers in these proceedings: • biomedical translation (Bawden et al., 2019b) • automatic post-editing (Chatterjee et al., 2019) • metrics (Ma et al., 2019) • quality estimation (Fonseca et al., 2019) • parallel corpus filtering (Koehn et al., 2019) • robustness (Li et al.,…”
COMET: A Neural Framework for MT Evaluation (2020, methodology): “5 The evaluation method used is the official Kendall’s Tau-like formulation, τ , from the WMT 2019 Metrics Shared Task (Ma et al., 2019) defined as:”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “We obtain the source language sentences, machine-translated texts and reference texts from the WMT19 metrics shared task [44].”
Results of the WMT20 Metrics Shared Task (2020, background): “The results conﬁrm the trends from previous years, namely metrics based on word or sentence-level embeddings, achieve the highest performance (Ma et al., 2018, 2019).”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “…also closely related to segment -level and system -level correlations to human judgments respectively, which have been extensively used in developing evaluation metrics in machine translation (Ma et al., 2019; Thompson and Post, 2020) and summarization (Bhandari et al., 2020; Deutsch et al., 2021).”
Decoding and Diversity in Machine Translation (2020, background): “, 2002), although several studies have questioned this correspondence (Callison-Burch et al., 2006; Ma et al., 2019).”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, unknown): “We compute chrF++ (Popovi´c, 2017) using both SacreBLEU and the scripts used by authors of the baseline systems.”
Machine Translation into Low-resource Language Varieties (2021, methodology): “We sample 10M EN-RU sentences from the WMT’19 shared task (Ma et al., 2019), and 80M RU sentences from the CoNLL’17 shared task to train embeddings.”
A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods (2021, methodology): “Since 2008, the Workshop on Machine Translation (WMT) has run a shared-task for developing evaluation metrics (Mathur et al., 2020, among others).”
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing (2020, result): “…always gauged the trustworthiness of automatic evaluation metrics in terms of how well, and how consistently, they correlate with human evaluation scores (Over et al., 2007; Gatt and Belz, 2008; Bojar et al., 2016; Shi-morina et al., 2018; Ma et al., 2019; Mille et al., 2019; Duˇsek et al., 2020).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “The WMT shared metrics competition (Ma et al., 2018, 2019; Mathur et al., 2020) is the main submission entry for evaluation metrics.”