COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task
By Ricardo Rei, José G. C. de Souza, Duarte M. Alves, Chrysoula Zerva, Ana C. Farinha, T. Glushkova, Alon Lavie, Luísa Coheur, André F. T. Martins. Conference on Machine Translation, 2022
503 citations recorded by the source corpus.
In this paper, we present the joint contribution of Unbabel and IST to the WMT 2022 Metrics Shared Task. Our primary submission – dubbed COMET-22 – is an ensemble between a COMET estimator model trained with Direct Assessments and a newly proposed multitask model trained to predict sentence-level scores along with OK/BAD word-level tags derived from Multidimensional Quality Metrics error annotations. These models are ensembled together using a hyper-parameter search that weights different features extracted from both evaluation models and combines them into a single score. For the reference-free evaluation, we present CometKiwi. Similarly to our primary submission, CometKiwi is an ensemble between two models. A traditional predictor-estimator model inspired by OpenKiwi and our new multitask model trained on Multidimensional Quality Metrics which can also be used without references. Both our submissions show improved correlations compared to state-of-the-art metrics from last year as well as increased robustness to critical errors.
Subfields: Artificial Intelligence, Information Systems
View Paper: https://www.semanticscholar.org/paper/5f0a1418494c5c8cc669fbf3a3a8395020122a2a
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Tom Kocmi
- Christian Federmann
- Luísa Coheur
- Ricardo Rei
- Amr Sharaf
- Mohamed Afify
- Hitokazu Matsushita
- Mohamed Gabr
In Information Systems
Citations in the corpus
- How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation (2023, methodology): “Specifically, we use COMET-22 (wmt22-COMET-da) (Rei et al., 2022a), a reference-based metric that combines direct assessments (DA), sentence-level scores, and word-level tags from Multidimensional Quality Metrics (MQM) error annotations.”
- Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023, methodology): “Effectively, we compare GEMBA against the best-performing automatic metrics: COMET (Rei et al., 2020, 2022), BLEURT (Sellam et al., 2020), or the non-public winner MetricX XXL.”
- xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “We benchmark our metrics widely used open neural metrics: COMET22 (Rei et al., 2022a)13 and BLEURT-20 (Pu et al.”