Models of Translation Competitions

By Mark Hopkins, Jonathan May. 2013

20 citations recorded by the source corpus.

Subfields: Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “Re-orderings of this type have been used to argue against the reliability of the ofﬁcial WMT ranking (Lopez, 2012; Hopkins and May, 2013).”
Efficient Elicitation of Annotations for Human Evaluation of Machine Translation (2014, methodology): “In the graphical model formulation of Hopkins and May (2013), the pairwise judgments ( i, S 1 , S 2 , π ) are imagined to have been generated according to the following process: • Select a source sentence i • Select two systems S 1 and S 2 .”
Can machine translation systems be evaluated by the crowd alone (2015, background): “A range of methods for going from sentence-level human assessments to a totally-ordered system ranking have been proposed (Bojar, Ercegovcevic, and Popel, 2011; Lopez, 2012; Callison-Burch et al., 2012; Hopkins and May, 2013), with no consensus on the best aggregation method.”
Squibs: Evaluating Human Pairwise Preference Judgments (2015, methodology): “…generally cited models for pairwise comparisons are the Thurstone model (Thurstone 1927) and the closely related Bradley-Terry (BT) model (Bradley and Terry 1952); these have connections to the IRT models, widely used in analyzing responses to questionnaires, which Hopkins and May (2013) drew on.”
Comparing Test Sets with Item Response Theory (2021, methodology): “Hopkins and May (2013) use IRT to estimate the relative ability of a set of machine translation systems using responses from pairwise comparison of system outputs by human judges.”
Ten Years of WMT Evaluation Campaigns: Lessons Learnt (2016, methodology): “This point was raised by Hopkins and May (2013), who then provided a Bayesian model formulation of the human ranking problem, which allowed them to use perplexity to compare different system rankings.”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “, 2019): machine translation (Hopkins and May, 2013) and chatbot evaluation (Sedoc”