Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

By Markus Freitag, George F. Foster, David Grangier, Viresh Ratnakar, Qijun Tan, Wolfgang Macherey. 2021

534 citations recorded by the source corpus.

Abstract Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Similarly, Freitag et al. (2021a) demonstrate that non-expert annotations often lead to mistaken claims of super-human model performance, when expert annotators correctly identify issues in the generated texts.”
Learning Compact Metrics for MT (2021, neutral): “These metrics deliver superior performance over those based on lexical overlap, outper-forming even crowd-sourced annotations (Freitag et al., 2021; Mathur et al., 2020b).”
GPTScore: Evaluate as You Desire (2023, methodology): “We consider a subdatasets of Multidimensional Quality Metrics (MQM) (Freitag et al., 2021), namely, MQM-2020 (Chinese->English).”
PaLM 2 Technical Report (2023, methodology): “MQM represents the average errors per segment, with lower numbers indicating better results.”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, unknown): “We use Freitag et al. (2021a) annotated TED dataset as our development set to select the hyper-parameters in Error Synthesis Models and SEScore Metric Model.”
Dialect-robust Evaluation of Generated Text (2022, methodology): “, 2017) and MQM (Freitag et al., 2021) which both rely on machine translated outputs.”
One Embedder, Any Task: Instruction-Finetuned Text Embeddings (2022, methodology): “, 2021), and WMT21 Chinese-to-English translation (Barrault et al., 2020; Freitag et al., 2021).”
Toxicity in Multilingual Machine Translation at Scale (2022, methodology): “Automatic measures compare the translation output to one or more human references, e.g., Papineni et al. (2002); Popovi´c (2015a); Lo (2019); Rei et al. (2020); Sellam et al. (2020); Freitag et al. (2021), or pre-trained embeddings, e.g., Lo (2019); Yankovskaya et al. (2019).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, background): “This is in line with the ﬁndings of Freitag et al. (2021), who showed that in some cases, metrics may be better than non-expert annotators.”
Building Machine Translation Systems for the Next Thousand Languages (2022, unknown): “Freitag et al. (2021) explores some of these phenomena more.”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “This step was necessary as Freitag et al. (2021a) showed that the DA-based ground-truth is already of lower quality than some of our submissions (Section 3).”
What happens if you treat ordinal ratings as interval data? Human evaluations in NLP are even more under-powered than you think (2021, result): “Similar studies have been conducted for machine translation (Freitag et al., 2021).”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “Taking the inspiration from (Freitag et al., 2021a), we obtain our pretraining signals aligning with human grading process.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “This conﬁrms the ﬁndings in Freitag et al. (2021) that when human evaluation is conducted by professional translators and MQM, high-quality human translations typically still outperform MT.”