Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance

By Qingsong Ma, Ondrej Bojar, Yvette Graham. 2018

105 citations recorded by the source corpus.

This paper presents the results of the WMT18 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT18 News Translation Task with automatic metrics. We collected scores of 10 metrics and 8 research groups. In addition to that, we computed scores of 8 standard metrics (BLEU, SentBLEU, chrF, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system-level correlation (how well each metric’s scores correlate with WMT18 official manual ranking of systems) and in terms of segment-level correlation (how often a metric agrees with humans in judging the quality of a particular sentence relative to alternate outputs). This year, we employ a single kind of manual evaluation: direct assessment (DA).

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Oftentimes these metrics vary in their effectiveness across languages; WMT shared tasks (Ma et al., 2018b) report that the speciﬁc language, dataset, and system signiﬁcantly affect which metric has the strongest correlation with human ratings.”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “The following year also abandoned HUME and fully relied on DA (Ma et al., 2018), and embeddingbased metrics started strongly outperforming other metrics.”
Learning Compact Metrics for MT (2021, positive): “1 Following established approaches (Ma et al., 2018, 2019), we utilize the human ratings from the previous years’ shared tasks for training.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
BERTScore: Evaluating Text Generation with BERT (2019, background): “The WMT18 dataset (Ma et al., 2018) contains outputs of 159 translation systems on 14 language pairs.”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “In the former case, we use the the best-performing contestants for each year, that is, chrF++ , BEER , Meteor++ , RUSE , Yisi1 , ESIM and Yisi1-SRL (Mathur et al., 2019).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “, 2018), • metrics (Ma et al., 2018), • quality estimation (Specia et al.”
COMET: A Neural Framework for MT Evaluation (2020, background): “For DA, it is common practice to convert scores into relative rankings ( DA RR) when the number of annotations per segment is limited (Bojar et al., 2017b; Ma et al., 2018, 2019).”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “For the machine translation task, due to the more expensive computational cost brought by larger text sets, we ﬁrst use WMT18 [43] as a development set to search for one best prompt and obtain the phrase “ Such as ”, which is then used for the test language pairs.”
Results of the WMT20 Metrics Shared Task (2020, methodology): “To strengthen the conclusions of our evaluation, in past years (Bojar et al., 2016, 2017; Ma et al., 2018), we included significance test results for large hybrid-super-samples of systems 10K hybrid systems were created per language pair, with corresponding DA human assessment scores by sampling pai”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “These evaluations are much larger than what was previously done in the ﬁeld, e.g., 100 instances for MT in Ma et al. (2018) or around 100 instances for summarization in Kim et al. (2019) and Hardy et al. (2019).”
NUBIA: NeUral Based Interchangeability Assessor for Text Generation (2020, methodology): “are not included in the segment-level evaluation Ma et al. (2018). In that setting, metrics are scored on their ability to preserve the human ranking using the Kendall’s Tau correlation coefficient.”
Unsupervised Quality Estimation for Neural Machine Translation (2020, methodology): “DA methodology is currently used for manual evaluation of MT quality at the WMT translation tasks, as well as for assessing the performance of reference-based automatic MT evaluation metrics at the WMT Metrics Task (Bojar et al., 2016, 2017; Ma et al., 2018, 2019).”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We use Kendall’s tau-like correlation, typically used for DA RR (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020b), for evaluating the primary submissions on the challenge sets.”
YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources (2019, methodology): “BLEU (Papineni et al., 2002) has long been shown not to correlate well with human judgment on translation quality (Machacek and Bojar, 2014; Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “The WMT shared metrics competition (Ma et al., 2018, 2019; Mathur et al., 2020) is the main submission entry for evaluation metrics.”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, methodology): “For evaluating MT systems, we use the data from the Metrics Shared Task in WMT 2018 and 2019 (Ma et al., 2018, 2019).”