Accurate Evaluation of Segment-level Machine Translation Metrics

By Yvette Graham, Timothy Baldwin, Nitika Mathur. 2015

116 citations recorded by the source corpus.

Evaluation of segment-level machine translation metrics is currently hampered by: (1) low inter-annotator agreement levels in human assessments; (2) lack of an effective mechanism for evaluation of translations of equal quality; and (3) lack of methods of significance testing improvements over a baseline. In this paper, we provide solutions to each of these challenges and outline a new human evaluation methodology aimed specifically at assessment of segment-level metrics. We replicate the human evaluation component of WMT-13 and reveal that the current state-of-the-art performance of segment-level metrics is better than previously believed. Three segment-level metrics — METEOR, NLEPOR and SENTBLEUMOSES — are found to correlate with human assessment at a level not significantly outperformed by any other metric in both the individual language pair assessment for Spanish-toEnglish and the aggregated set of 9 language pairs.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “While the shortcomings of these metrics are well known for MT (Graham, 2015; Espinosa et al., 2010; Cahill, 2009), the problem is aggravated for dialog generation evaluation because of the much larger output space (Liu et al., 2016).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, background): “It creates an explicit alignment between the candidate and target responses.”
Neural Approaches to Conversational AI (2018, neutral): “orrelation analysis was performed at the sentence level, but decent sentence-level correlation has long been known to be difﬁcult to achieve even for machine translation (Callison-Burch et al., 2009; Graham et al., 2015), the task for which the underlying metrics (e.g., BLEU and METEOR) were speciﬁc”
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets (2015, background): “…not immune from criticism (e.g., Callison-Burch et al. (2006)), its properties are well understood, B LEU scores have been shown to correlate well with human judgments (Doddington, 2002; Coughlin, 2003; Graham and Baldwin, 2014; Graham et al., 2015) in SMT, and it has allowed the ﬁeld to proceed.”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, background): “odel. 6.2 Human Evaluation Itiswell known that automatic evaluation metrics such as BLEU encourage reference-like translations and do not fully capture true translation quality (Callison-Burch, 2009; Graham et al., 2015). Therefore, we also carry out a recently proposed evaluation from (Graham et al”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, background): “DA scores for systems and segments have been shown to be highly repeatable in self-replication experiments (Graham et al., 2015).”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “A single form of golden truth of translation quality judgement is used this year: • In Direct Assessment (DA) (Graham et al., 2016), humans assess the quality of a given MT output translation by comparison with a reference translation (as opposed to the source and reference).”
Results of the WMT16 Metrics Shared Task (2016, methodology): “Segment-level DA Adequacy assessments were collected for translations sampled from the output of systems participating in WMT16 translation task for seven language pairs (Graham et al., 2015).”
Results of the WMT17 Metrics Shared Task (2017, methodology): “Two sources of golden truth of translation quality judgement are used this year: • In Direct Assessment (DA) (Graham et al., 2015), humans assess the quality of a given MT output translation by comparison with a reference translation (but not the source).”
Results of the WMT20 Metrics Shared Task (2020, methodology): “Since we do not have a total order ranking of all translations, it is not possible to apply conventional Kendall’s Tau given the current DARR human evaluation setup (Graham et al., 2015).”
Unsupervised Quality Estimation for Neural Machine Translation (2020, methodology): “The current practice in MT evaluation is the so called Direct Assessment (DA) of MT quality (Graham et al., 2015b), where raters evaluate the MT on a continuous 1-100 scale.”
Can machine translation systems be evaluated by the crowd alone (2015, background): “For evaluations to be accurate at the segment-level, a different approach is required, where assessments are repeated per segment as opposed to per system; see Graham et al. (2015) for further details.”
Re-evaluating Automatic Metrics for Image Captioning (2016, background): “The test has shown to be valuable for evaluation of document and segment-level machine translation (Graham and Baldwin, 2014; Graham et al., 2015; Graham and Liu, 2016) and summarization metrics (Graham, 2015).”
On Evaluating and Comparing Open Domain Dialog Systems (2018, background): “There is work in MT and Natural Language Generation (NLG) fields which studies correlation of BLEU with human judgement and highlights some of its shortcomings (Graham, 2015; Espinosa et al., 2010; Cahill, 2009).”
Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models (2017, background): “We do not report BLEU scores for individual users, as the dev and test set for each specific user tends to be small (less than 500 samples) and BLEU is known to be unreliable when evaluated on small datasets (Graham et al., 2015; Liu et al., 2016).”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, methodology): “To ensure that observed improvement is statistically significant we follow common consensus (Deutsch et al., 2021; Graham, 2015; Graham et al., 2015; Graham and Baldwin, 2014) in NMT and rely on William test (Steiger, 1980) as considered observations are correlated3.”
Sequence Effects in Crowdsourced Annotations (2017, methodology): “In this paper, we base our analysis on the adequacy dataset of Graham et al. (2015), on Spanish-English newswire data from WMT 2013 (Bojar et al., 2013).”