Efficient Elicitation of Annotations for Human Evaluation of Machine Translation

By Keisuke Sakaguchi, Matt Post, Benjamin Van Durme. 2014

87 citations recorded by the source corpus.

A main output of the annual Workshop on Statistical Machine Translation (WMT) is a ranking of the systems that participated in its shared translation tasks, produced by aggregating pairwise sentencelevel comparisons collected from human judges. Over the past few years, there have been a number of tweaks to the aggregation formula in attempts to address issues arising from the inherent ambiguity and subjectivity of the task, as well as weaknesses in the proposed models and the manner of model selection. We continue this line of work by adapting the TrueSkill TM algorithm — an online approach for modeling the relative skills of players in ongoing competitions, such as Microsoft’s Xbox Live — to the human evaluation of machine translation output. Our experimental results show that TrueSkill outperforms other recently proposed models on accuracy, and also can significantly reduce the number of pairwise annotations that need to be collected by sampling non-uniformly from the space of system competitions.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “Sakaguchi et al. (2014) provide an adaptation of this approach to the WMT manual evaluation, and showed that it performed well on WMT13 data.”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, methodology): “The final evaluation results were produced using the TrueSkill algorithm (Herbrich et al., 2006; Sakaguchi et al., 2014).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “From these rankings, we produced pairwise translation comparisons, and applied the TrueSkill algorithm (Herbrich et al., 2007; Sakaguchi et al., 2014) to produce system rankings.”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “, 2006), are therefore applied by recent MT evaluation studies (Sakaguchi et al., 2014; Bojar et al., 2016) to produce overall system rankings based on a sample of binary comparisons.”
Findings of the E2E NLG Challenge (2018, methodology): “The final evaluation results were produced using the TrueSkill algorithm (Herbrich et al., 2006; Sakaguchi et al., 2014), with partial ordering into significance clusters computed using bootstrap resampling (Bojar et al.”
End-to-End Content and Plan Selection for Data-to-Text Generation (2018, methodology): “In order to gather ranks for the naturalness, generations were shown without the meaning representation and rated based on how likely an utterance could have been produced by a na-# (2014).”
A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation (2018, methodology): “The final scores were produced using the TrueSkill algorithm (Sakaguchi et al., 2014) through pairwise comparisons of the human evaluation scores among the 20 competing systems.”
Ground Truth for Grammatical Error Correction Metrics (2015, methodology): “Each system’s quality was estimated and the total ranking was produced on this dataset using the TrueSkill model (Sakaguchi et al., 2014), as done in WMT 2014.”
Human Effort and Machine Learnability in Computer Aided Translation (2014, background): “Let S be the set of pairwise judgments and (5) Sakaguchi et al. (2014) showed that, despite its simplicity, Eq.”
TNT-NLG , System 2 : Data Repetition and Meaning Representation Manipulation to Improve Neural Generation (2018, methodology): “The TrueSkill algorithm was used to score the systems based on the annotations (Sakaguchi et al., 2014).”
JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correction (2017, methodology): “The absolute human ranking of systems was inferred using TrueSkill, which computes a relative score from pairwise comparisons, and we cluster systems with overlapping ranges into equivalence classes by bootstrap resampling (Sakaguchi et al., 2014; Herbrich et al., 2006).”
Ten Years of WMT Evaluation Campaigns: Lessons Learnt (2016, methodology): “The method that won was a new approach that based on the TrueSkill algorithm (Sakaguchi et al., 2014).”
Efficient Online Scalar Annotation with Bounded Support (2018, methodology): “This approach has been used in the NLP community in tasks such as for assessing machine translation quality (Bojar et al., 2014; Sakaguchi et al., 2014; Bojar et al., 2015, 2016) to collect pairwise judgments efficiently.”
NLG Challenge : Neural Models vs . Templates (2018, methodology): “The final evaluation results were produced by the TrueSkill algorithm (Sakaguchi et al., 2014), which performs pairwise system comparisons and clusters them into groups.”
Deep Literature Survey Automation with an Iterative Workflow (2025, unknown): “…an anonymized survey pair and asked to select the superior one based on multiple quality dimensions, including coverage, relevance, structural coherence, and overall quality, which is more objective and stable than ranking based on absolute scores (Herbrich et al., 2006; Sakaguchi et al., 2014).”