Putting Human Assessments of Machine Translation Systems in Order

By Adam Lopez. 2012

24 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “As discussed, the method used to compute the ranking has been tweaked a bit each year over the past few years in response to criticisms (e.g., Lopez (2012); Bojar et al. (2011)).”
Efficient Elicitation of Annotations for Human Evaluation of Machine Translation (2014, background): “Over the past few years, the WMT organizers have introduced a number of minor tweaks to the ranking algorithm ( § 2) in reaction to largely intuitive arguments that have been raised about how the evaluation is conducted (Bo-jar et al., 2011; Lopez, 2012).”
Simulating human judgment in machine translation evaluation campaigns (2012, methodology): “Lopez [4] proposed an alternative method to rank systems.”
Squibs: Evaluating Human Pairwise Preference Judgments (2015, background): “” Pado et al. (2009) modified the systemlevel predictions approach to become “tie-aware,” and noted that that this “makes a considerable practical difference, improving correlation figures by 5–10 points.” At around the same time Vilar et al. (2007) examined the use of pairwise comparisons in MT eva”
Findings of the 2012 Workshop on Statistical Machine Translation (2012, background): “This year another paper shows some additional potential inconsistencies in the rankings (Lopez, 2012).”
Models of Translation Competitions (2013, background): “Let’s revisit (Lopez, 2012)’s objection to the B O - JAR ranking heuristic: “...couldn’t a system still be penalized simply by being compared to [good systems] more frequently than its competitors?””
Ten Years of WMT Evaluation Campaigns: Lessons Learnt (2016, methodology): “In Proc. of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1416–1424, Sofia, Bulgaria.”
Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation (2019, background): “Human assessments also need to be consistent and repeatable for a meaningful evaluation (Lopez, 2012).”