Can machine translation systems be evaluated by the crowd alone

By Yvette Graham, Timothy Baldwin, Alistair Moffat, Justin Zobel. 2015

206 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “In addition to the standard relative ranking (RR) manual evaluation, this year a new method of human evaluation was also trialed in the main translation task: monolingual direct assessment (DA) of translation fluency (Graham et al., 2013) and adequacy (Graham et al., 2014, 2016).”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “While these eﬀects can be lessened using multiple references, the underlyingproblemremainsunsolved3. Therefore, following the Conference on Machine Translation (WMT17) [6], we adopt direct assessment [17]asourhumanevaluationmethod. Toavoidreferencebias—whichcanalsohappen forhumanevaluation4—weusethe”
Fully Character-Level Neural Machine Translation without Explicit Segmentation (2016, methodology): “Therefore, we also carry out a recently proposed evaluation from (Graham et al., 2016) where we have human assessors rate both (1) adequacy and (2) ﬂuency of each system translation on a scale from 0 to 100 via Amazon Mechanical Turk.”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “Work on evaluation over the past few years has provided fresh insight into ways to collect direct assessments (DA) of machine translation quality (Graham et al., 2013, 2014, 2016), and two years ago the evaluation campaign included parallel assessment of a subset of News task language pairs…”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “…the APE systems and produce a ranking based on human judgment, as well as analyze how humans perceive TER/BLEU performance differences between the submitted systems, a human evaluation of the quality of automatic post-edits was carried out using Direct Assessment (DA) (Graham et al., 2013, 2016).”
Findings of the 2019 Conference on Machine Translation (WMT19) (2019, background): “Work on evaluation over the past few years has provided fresh insight into ways to collect direct assessments (DA) of machine translation quality (Graham et al., 2013, 2014, 2016), and three years ago the evaluation campaign included parallel assessment of a subset of News task language pairs…”
Results of the WMT16 Metrics Shared Task (2016, neutral): “, 2013) and adequacy (Graham et al., 2014; Graham et al., 2016).”
On Evaluating and Comparing Conversational Agents (2018, background): “In other directions, Graham et al. (2017) try to use crowdsourced workers to evaluate MT systems but there is no consensus on “good human evaluation” and reliably getting such evaluations can be prohibitively expensive.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “Direct Assessment (DA, Graham et al., 2013, 2014a, 2016) was employed as the source of the “golden truth” to evaluate metrics again this year.”
Findings of the Third Shared Task on Multimodal Machine Translation (2018, methodology): “The system outputs indicated as the primary sub-mission were manually evaluated by bilingual Direct Assessment (DA) (Graham et al., 2015) using the Appraise platform (Federmann, 2012).”
Results of the WMT17 Metrics Shared Task (2017, methodology): “Direct Assessment (DA) This year the translation task employed monolingual direct assessment (DA) of translation adequacy (Graham et al., 2013; Graham et al., 2014; Graham et al., 2016).”
COMET: A Neural Framework for MT Evaluation (2020, background): “These DAs are then mapped into relative rankings ( DA RR) (Ma et al., 2019).”
BLEU Might Be Guilty but References Are Not Innocent (2020, background): “With similar cost motivations, previous work has advocated for contracting evaluation to crowd workers instead of language experts (Goto et al., 2014; Graham et al., 2017).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “For each output, they are usually asked to check two important criteria: fluency and adequacy of the hypothesis [56] as described below.”
Enhancing AMR-to-Text Generation with Dual Graph Representations (2019, unknown): “We employ the Direct Assessment (DA) method (Graham et al., 2017) via Amazon Mechanical Turk. Using the DA method inspired by Mille et al. (2018), we assess two quality criteria: (i) meaning similarity: how close in meaning the generated text is to the gold sentence; and (ii) readability: how well t”
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (2017, methodology): “…fall under either the phrase-based or neural approaches, were evaluated (i) automatically with the BLEU (Papineni et al., 2002) and TER (Snover et al., 2006) metrics, and (ii) manually by means of ranking translations (Federmann, 2012) and monolingual semantic similarity (Graham et al., 2016).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, background): “Since running a comparison of direct assessments (DA, Graham et al., 2013, 2014, 2016) and relative ranking in 2016 (Bojar et al., 2016) and verifying a high correlation of system rankings for the two methods, as well as the advantages of DA, such as quality controlled crowd-sourcing and linear…”