Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics

By A. Lommel, Hans Uszkoreit, Aljoscha Burchardt. 2014

En els ultims anys l'avaluacio de la qualitat de la traduccio s'ha convertit en un tema rellevant i a la vegada que, de vegades, polemic. La perspectiva de la industria sobre la qualitat esta altament fragmentada, en part perque diferents tipus de projectes de traduccio requereixen metodes molt diferents d'avaluacio. A mes, els metodes d'avaluacio de la qualitat de les traduccions humanes i de les traduccions elaborades amb traduccio automatica (TA) son d'indole diferent. La manca de claredat provoca incertesa sobre si una traduccio compleix amb les necessitats del seu promotor o l'usuari final, i deixa als proveidors amb dubtes sobre el que els clients volen i necessiten. Com a resposta a aquest fet, el projecte QTLaunchPad, financat per la Unio Europea, ha desenvolupat el marc denominat Multidimensional Quality Metrics (MQM), un sistema obert i ampliable per declarar i descriure les metriques sobre qualitat en traduccio utilitzant un vocabulari compartit de "classes de problemes”.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “The latest evaluations thus followed the suggestion by Freitag et al. (2021a) to use Multidimensional Quality Metrics (MQM, Lommel et al., 2014), a fine-grained expert-based annotation approach.”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, background): “eu) to address the shortcomings of previous quality evaluation methods (Lommel et al., 2014).”
COMET: A Neural Framework for MT Evaluation (2020, methodology): “The MQM metric referred to throughout this paper is an internal metric deﬁned in accordance with the MQM framework (Lommel et al., 2014) (MQM).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “Additionally, due to technical restrictions on the length of an evaluation session, we limited the MQM evaluation to the first 12 segments per document.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “…implemented several changes to the methodology that was followed in previous years editions of the task: • Expert-based human evaluation This year, we collected our own human ratings for selected language pairs (en → de, en → ru, zh → en) from professional translators via MQM (Lommel et al., 2014).”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “The Multidimensional Quality Metrics (MQM) framework (Lommel et al., 2014), on the other hand, offers a more comprehensive and systematic approach to MT evaluation.”
Style Over Substance: Evaluation Biases for Large Language Models (2023, unknown): “More recent work in MT evaluation also proposes evaluation broken down into more aspects (Lommel et al., 2014).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “We implemented several changes to the methodology that was followed in previous years’ editions: • Expert-based human evaluation : Like last year, we collected our own human ratings for select language pairs (en → de, en → ru, zh → en) from professional translators via MQM (Lommel et al., 2014).”