A Grain of Salt for the WMT Manual Evaluation

By Ondrej Bojar, Milos Ercegovcevic, Martin Popel, Omar Zaidan. 2011

69 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, background): “…assessment DA employs also allows the sentence length restriction to be removed.8 DA also aims to avoid the possible source of bias identified in Bojar et al. (2011), introduced by simultaneous assessment of several translations at once, where systems for which translations were more frequently…”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “As discussed, the method used to compute the ranking has been tweaked a bit each year over the past few years in response to criticisms (e.g., Lopez (2012); Bojar et al. (2011)).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “When evaluating competing translations of the same source that are situated in close proximity within a HIT, annotators may be primed by high (or low) quality outputs resulting in overly severe (or lenient) judgments for subsequent translations of the same source segment (Bojar et al., 2011).”
Results of the WMT15 Metrics Shared Task (2015, methodology): “For the metrics task in 2014, we were still using the “Pre-TrueSkill” method called “ > Others”, see Bojar et al. (2011).”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, background): “In addition, DA avoids bias that has been problematic in previous evaluations introduced by assessment of several alternate translations on a single screen, where scores for translations had been unfairly penalized if often compared to high quality translations (Bojar et al., 2011).”
Results of the WMT16 Metrics Shared Task (2016, methodology): “Furthermore, DA avoids bias that has been problematic in previous evaluations introduced by simultaneous assessment of several alternate translations of a given single source language input, where scores of systems for which translations were often compared to high or low quality translations result”
Results of the WMT17 Metrics Shared Task (2017, background): “Furthermore, DA avoids bias that has been problematic in previous evaluations introduced by assessment of several alternate translations on one screen, where scores for translations were unfairly penalized if often compared to high quality translations (Bojar et al., 2011).”
Can machine translation systems be evaluated by the crowd alone (2015, methodology): “For example, if a translation happens to appear in the same displayed set as a high quality translation, it is likely that its score will be pushed down (Bojar et al. 2011).”
Ten Years of WMT Evaluation Campaigns: Lessons Learnt (2016, methodology): “In recent years, we have used Sentence Ranking as the sole method of automatic evaluation (refining it according to certain criticisms (Bojar et al., 2011; Lopez, 2012; Koehn, 2012)), but ongoing problems with reliability, interpretability and poor scalability with increasing numbers of systems…”
Findings of the 2012 Workshop on Statistical Machine Translation (2012, methodology): “Last year one of the long papers published at WMT criticized our method for compiling the overall ranking for systems in the translation task (Bojar et al., 2011).”
Chimera – Three Heads for English-to-Czech Translation (2013, background): “Overall, TectoMT sometimes even ties with a highly tuned Moses configuration in manual evaluations, see Bojar et al. (2011).”
Adaptation of machine translation for multilingual information retrieval in the medical domain (2014, methodology): “[96], the output was transformed to pairwise comparison and is presented as a percentage”
Models of Translation Competitions (2013, background): “Lopez’s concern, while reasonable, is less obviously damning than (Bojar et al., 2011)’s criticism of O RIG WMT.”
Putting Human Assessments of Machine Translation Systems in Order (2012, methodology): “(2) However, the analysis by Bojar et al. (2011) goes further and suggests disregarding the effect of ties altogether by removing them from the denominator.”
Efficient Elicitation of Annotations for Human Evaluation of Machine Translation (2014, methodology): “Another of Bojar et al. (2011)’s suggestions was to exclude this data, because systems compared more often against the references suffered unfairly.”
Appraise: an Open-Source Toolkit for Manual Evaluation of MT Output (2012, methodology): “- Krippendorff’s α as described by Krippendorff (2004); - Fleiss’ κ as published in Fleiss (1971), extending work from Cohen (1960); - Bennett, Alpert, and Goldstein’s S as defined in Bennett et al.”
Is Machine Translation Getting Better over Time? (2014, background): “This has the added advantage of eliminating the criticism made of WMT evaluations that systems sometimes gain advantage from luckof-the-draw comparison with low quality output, and vice-versa (Bojar et al., 2011).”