Improving Evaluation of Machine Translation Quality Estimation

By Yvette Graham. 2015

51 citations recorded by the source corpus.

Quality estimation evaluation commonly takes the form of measurement of the error that exists between predictions and gold standard labels for a particular test set of translations. Issues can arise during comparison of quality estimation prediction score distributions and gold label distributions, however. In this paper, we provide an analysis of methods of comparison and identify areas of concern with respect to widely used measures, such as the ability to gain by prediction of aggregate statistics specific to gold label distributions or by optimally conservative variance in prediction score distributions. As an alternative, we propose the use of the unit-free Pearson correlation, in addition to providing an appropriate method of significance testing improvements over a baseline. Components of WMT-13 and WMT-14 quality estimation shared tasks are replicated to reveal substantially increased conclusivity in system rankings, including identification of outright winners of tasks.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “Statistical significance on Pearson r and Spearman rho was computed using the William’s test, following the approach suggested in (Graham, 2015).”
Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE (2015, methodology): “…the suitability of Williams signiﬁcance test (Williams, 1959) for evaluation of automatic MT metrics (Graham and Baldwin, 2014; Graham et al., 2015; Graham, 2015), and, for similar reasons, Williams test is suited to signiﬁcance testing differences in performance of competing summarization…”
Pushing the Limits of Translation Quality Estimation (2017, methodology): “For sentence-level QE, we report the Pearson’s r correlation for HTER prediction and the Spearman’s ρ correlation score for sentence ranking (Graham, 2015).”
Group MAD Competition? A New Methodology to Compare Objective Image Quality Models (2016, background): “l = [1, 2, 3, 4, 5, 6] represents the 6 distortion levels and qij is a 6 × 1 vector that contains the corresponding quality scores given by a model to 6 images, which have the same (i-th) source image and the same (j-th) distortion type but different distortion levels.”
Metrics for Evaluation of Word-level Machine Translation Quality Estimation (2016, background): “Graham (2015) shows that Pearson correlation better suits for the evaluation of sentence-level QE systems than mean absolute error (MAE), often used for this purpose.”