COMET: A Neural Framework for MT Evaluation

By Ricardo Rei, Craig Alan Stewart, Ana C. Farinha, Alon Lavie. 2020

1,636 citations recorded by the source corpus.

We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “In a survey of such studies by Reiter (2018) focused on BLEU, he concludes that it is useful as a diagnostic tool during the development of MT systems, but not for other tasks and that is should not be used at the segment level.”
Learning Compact Metrics for MT (2021, positive): “Models Like C OMET (Rei et al., 2020a) and B LEURT (Sellam et al., 2020a), we treat evaluation as a regression problem where, given a reference translation x (typically produced by a human) and predicted translation ˜ x (produced by an MT sys-tem), the goal is to predict a real-valued human rating…”
Beyond English-Centric Multilingual Machine Translation (2020, background): “Their quality must be carefully measured, in part with automatic metrics such as BLEU, COMET (Rei et al., 2020), and BLEURT (Sellam et al.”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, methodology): “More recently, Rei et al. (2020) used MQM labels to ﬁne-tune COMET for automatic evaluation.”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “, supervised metrics): COMET [47], BLEURT [52], or are human judgment-free (i.”
Results of the WMT20 Metrics Shared Task (2020, methodology): “COMET* metrics (Rei et al., 2020b) were build us-ing the Estimator model or the Translation Ranking model proposed in Rei et al. (2020a).”
Unbabel’s Participation in the WMT20 Metrics Shared Task (2020, methodology): “, 2020) and our more recent COMET (Rei et al., 2020), all build upon developments in language modelling to generate automatic metrics with high correlation”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, neutral): “…metrics (BLEU (Papineni et al., 2002), ChrF (Popovi´c, 2015) and TER (Snover et al., 2006)); (2) learned metrics requiring human rating data (COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020)); (3) learned metrics without human rating data (PRISM (Thompson and Post, 2020), BARTScore (Yuan et…”
Towards Making the Most of ChatGPT for Machine Translation (2023, methodology): “Hence, we adopt the mostly used COMET (Rei et al., 2020) as our primary metric.”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, background): “Concurrent work has investigated incorporating the source with great success (Rei et al., 2020).”
Toxicity in Multilingual Machine Translation at Scale (2022, methodology): “Automatic measures compare the translation output to one or more human references, e.g., Papineni et al. (2002); Popovi´c (2015a); Lo (2019); Rei et al. (2020); Sellam et al. (2020); Freitag et al. (2021), or pre-trained embeddings, e.g., Lo (2019); Yankovskaya et al. (2019).”
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand (2021, background): “• When only one reference is available per instance, COMET-QE (a strong referenceless metric with crosslingual contextual representations; Rei et al., 2020) achieves higher correlations with human judgments than all reference-based metrics.”
Prompting Large Language Model for Machine Translation: A Case Study (2023, methodology): “We evaluate translation performance using both a surface-based metric, detokenized BLEU↑ from SacreBLEU (Post, 2018), and a model-based metric, COMET↑ from unbabel-comet with wmt20comet-da (Rei et al., 2020).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, background): “Nevertheless, we observe that the metric rankings are largely stable across both granularities and that M ETRIC X XXL and C OMET -22 perform best on both the segment-level and system-level tasks.”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, background): “Notably, learned neural metrics, such as C OMET (Rei et al., 2020) and B LEURT (Sellam et al., 2020), have demonstrated significant improvements in terms of correlation with human judgments when compared to traditional metrics like BLEU (Papineni et al., 2002; Freitag et al., 2021b, 2022).”
Dialect-robust Evaluation of Generated Text (2022, methodology): “Although contemporary metrics such as COMET (Rei et al., 2020) can be reference-free, they still rely on training data and ∗Jiao and Tu are interns at Google. rater pools that do not cover all dialects of interest, leading to a high number of out-of-domain dialects.”
Reinforced Self-Training (ReST) for Language Modeling (2023, methodology): “We chose machine translation as a testbed for ReST as it is an impactful application of conditional language modeling where established reward models are available, for example, Metric X (Freitag et al., 2022), BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020).”
Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, methodology): “Recently, using pre-trained language models as NLG evaluation metric, e.g. , MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020), COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020), BARTScore (Yuan et al., 2021) and MAUVE (Pillutla et al., 2022), receives increasing attention since it”