xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection

By Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luísa Coheur, Pierre Colombo, André Martins. Transactions of the Association for Computational Linguistics, 2023

330 citations recorded by the source corpus.

Abstract Widely used learned metrics for machine translation evaluation, such as Comet and Bleurt, estimate the quality of a translation hypothesis by providing a single sentence-level score. As such, they offer little insight into translation errors (e.g., what are the errors and what is their severity). On the other hand, generative large language models (LLMs) are amplifying the adoption of more granular strategies to evaluation, attempting to detail and categorize translation errors. In this work, we introduce xcomet, an open-source learned metric designed to bridge the gap between these approaches. xcomet integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation (sentence-level, system-level, and error span detection). Moreover, it does so while highlighting and categorizing error spans, thus enriching the quality assessment. We also provide a robustness analysis with stress tests, and show that xcomet is largely capable of identifying localized critical errors and hallucinations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/871b196639337f5d5b7da2ae9d747583a1bc484e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yunmo Chen
  2. Amr Sharaf
  3. Haoran Xu
  4. Lingfeng Shen
  5. Young Jin Kim
  6. Kenton Murray
  7. Weiting Tan
  8. Benjamin Van Durme

In Computer Vision and Pattern Recognition

  1. Amr Sharaf
  2. Young Jin Kim
  3. Benjamin Van Durme

Citations in the corpus