Unbabel’s Participation in the WMT20 Metrics Shared Task

By Ricardo Rei, Craig Alan Stewart, Catarina Farinha, Alon Lavie. 2020

91 citations recorded by the source corpus.

We present the contribution of the Unbabel team to the WMT 2020 Shared Task on Metrics. We intend to participate on the segmentlevel, document-level and system-level tracks on all language pairs, as well as the “QE as a Metric” track. Accordingly, we illustrate results of our models in these tracks with reference to test sets from the previous year. Our submissions build upon the recently proposed COMET framework: we train several estimator models to regress on different humangenerated quality scores and a novel ranking model trained on relative ranks obtained from Direct Assessments. We also propose a simple technique for converting segment-level predictions into a document-level score. Overall, our systems achieve strong results for all language pairs on previous test sets and in many cases set a new state-of-the-art.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Learning Compact Metrics for MT (2021, neutral): “Popular examples of those metrics include C OMET (Rei et al., 2020a) and B LEURT -”
Results of the WMT20 Metrics Shared Task (2020, methodology): “COMET* metrics (Rei et al., 2020b) were build us-ing the Estimator model or the Translation Ranking model proposed in Rei et al. (2020a).”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “• COMET-DA_2020 is the same model submitted for last year’s shared task (Rei et al., 2020b; Mathur et al., 2020b) while COMET-DA_2021 is a retrained version of the previous model that includes the DA udgements collected in 2020.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “For that purpose we used three different systems: a C OMET Estimator (Rei et al., 2020a) trained on DA, a newly proposed Sequence Tagger, trained with MQM data, that works with and without references, and a C OMET K IWI (Rei et al., 2022) model trained on DA.”