Continuous Measurement Scales in Human Evaluation of Machine Translation

By Yvette Graham, Timothy Baldwin, Alistair Moffat, Justin Zobel. 2013

267 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “In addition to the standard relative ranking (RR) manual evaluation, this year a new method of human evaluation was also trialed in the main translation task: monolingual direct assessment (DA) of translation fluency (Graham et al., 2013) and adequacy (Graham et al.”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “Work on evaluation over the past few years has provided fresh insight into ways to collect direct assessments (DA) of machine translation quality (Graham et al., 2013, 2014, 2016), and two years ago the evaluation campaign included parallel assessment of a subset of News task language pairs…”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, background): “Recent studies show that continuous scales allow subjects to give more nuanced judgements (Belz and Kow, 2011; Graham et al., 2013; Bojar et al., 2017a).”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “…the APE systems and produce a ranking based on human judgment, as well as analyze how humans perceive TER/BLEU performance differences between the submitted systems, a human evaluation of the quality of automatic post-edits was carried out using Direct Assessment (DA) (Graham et al., 2013, 2016).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Instead of a direct assessment (Graham et al., 2013) often used in benchmark MT evaluations (Barrault et al., 2019; Guzm´an et al., 2019), we opt for post-editing.”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “Direct Assessment (DA) This year the translation task employed monolingual direct assessment (DA) of translation adequacy (Graham et al., 2013; Graham et al., 2014a; Graham et al., 2016).”
Results of the WMT16 Metrics Shared Task (2016, methodology): “Direct Assessment (DA) In addition to the standard relative ranking (RR) manual evaluation employed to yield official system rankings in WMT16 translation task, this year the translation task also trialed a new method of human evaluation, monolingual direct assessment (DA) of translation fluency (Gr”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, methodology): “The HITS were structured to include degraded translations and repeat translations, and rated on a continuous Likert scale with a single translation assessment displayed to the assessor at one time (Graham et al., 2014a; Graham et al., 2013).”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “We followed a setup similar to direct assessment (Graham et al., 2013).”
RankME: Reliable Human Ratings for Natural Language Generation (2018, methodology): “ etc. Note that ME implies the use of continuous scales, i.e. rating scales without numerical labels, similar to the visual analogue scales used by Belz and Kow (2011) or direct assessment scales of (Graham et al., 2013; Bojar et al., 2017), however, without given end-points. Siddharthan and Katsos ”
Results of the WMT17 Metrics Shared Task (2017, methodology): “Direct Assessment (DA) This year the translation task employed monolingual direct assessment (DA) of translation adequacy (Graham et al., 2013; Graham et al., 2014; Graham et al., 2016).”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, methodology): “Our SQM experiments used the 0-6 rating scale described above, instead of the wider, continuous scale recommended by (Graham et al., 2013), as this scale has been an established part of our existing MT evaluation ecosystem.”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, methodology): “For translation quality assessment, we followed a setup similar to direct assessment (Graham et al., 2013).”
COMET: A Neural Framework for MT Evaluation (2020, methodology): “…et al., 2019; Conneau and Lample, 2019; Conneau et al., 2019) to generate prediction estimates of human judgments such as Direct Assessments (DA) (Graham et al., 2013), Human-mediated Translation Edit Rate (HTER) (Snover et al., 2006) and metrics compliant with the Multidimensional Quality…”
Results of the WMT20 Metrics Shared Task (2020, background): “Thanks to the fact that the system-level DA score is an average over many judgments, mean scores are consistent and have been found to be reproducible (Graham et al., 2013).”
Unbabel’s Participation in the WMT20 Metrics Shared Task (2020, methodology): “…model against different types of human judgement; Human-mediated Translation Edit Rate (HTER) (Snover et al., 2006), a proprietary implementation of Multidimensional Quality Metric (MQM) (Lommel et al., 2014) and (in-line with the present task) Direct Assessments (DA) (Graham et al., 2013).”