Is Machine Translation Getting Better over Time?

By Yvette Graham, Timothy Baldwin, Alistair Moffat, Justin Zobel. 2014

65 citations recorded by the source corpus.

Recent human evaluation of machine translation has focused on relative preference judgments of translation quality, making it difficult to track longitudinal improvements over time. We carry out a large-scale crowd-sourcing experiment to estimate the degree to which state-of-theart performance in machine translation has increased over the past five years. To facilitate longitudinal evaluation, we move away from relative preference judgments and instead ask human judges to provide direct estimates of the quality of individual translations in isolation from alternate outputs. For seven European language pairs, our evaluation estimates an average 10-point improvement to state-of-theart machine translation between 2007 and 2012, with Czech-to-English translation standing out as the language pair achieving most substantial gains. Our method of human evaluation offers an economically feasible and robust means of performing ongoing longitudinal evaluation of machine translation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “In addition to the standard relative ranking (RR) manual evaluation, this year a new method of human evaluation was also trialed in the main translation task: monolingual direct assessment (DA) of translation fluency (Graham et al., 2013) and adequacy (Graham et al., 2014, 2016).”
Findings of the 2018 Conference on Machine Translation (WMT18) (2018, background): “Work on evaluation over the past few years has provided fresh insight into ways to collect direct assessments (DA) of machine translation quality (Graham et al., 2013, 2014, 2016), and two years ago the evaluation campaign included parallel assessment of a subset of News task language pairs…”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, background): “Work on evaluation over the past few years has provided fresh insight into ways to collect direct assessments (DA) of machine translation quality (Graham et al., 2013, 2014, 2016), and last year’s evaluation campaign included parallel assessment of a subset of News task language pairs evaluated…”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “Direct Assessment (DA) This year the translation task employed monolingual direct assessment (DA) of translation adequacy (Graham et al., 2013; Graham et al., 2014a; Graham et al., 2016).”
Results of the WMT16 Metrics Shared Task (2016, neutral): “, 2013) and adequacy (Graham et al., 2014; Graham et al., 2016).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, methodology): “…• English-to-Russian: 27% Although signiﬁcance tests for evaluation of MT systems and document-level metrics have been iden-tiﬁed (Koehn, 2004; Graham and Baldwin, 2014; Graham et al., 2014b), no such test has been proposed for segment-level metrics, and it is unfortunately common to conclude…”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “Direct Assessment (DA, Graham et al., 2013, 2014a, 2016) was employed as the source of the “golden truth” to evaluate metrics again this year.”
Results of the WMT17 Metrics Shared Task (2017, methodology): “Direct Assessment (DA) This year the translation task employed monolingual direct assessment (DA) of translation adequacy (Graham et al., 2013; Graham et al., 2014; Graham et al., 2016).”
COMET: A Neural Framework for MT Evaluation (2020, background): “These DAs are then mapped into relative rankings ( DA RR) (Ma et al., 2019).”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, methodology): “We have employed Direct Assessment (DA, Graham et al., 2013, 2014, 2016) as the primary mechanism for evaluating systems since running a comparison of DA and relative ranking in 2016 (Bojar et al., 2016).”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, background): “Human evaluation for machine translation (Graham et al., 2013, 2014; Sakaguchi and Van Durme, 2018) typically involves crowdsourcing where qualiﬁed crowd workers score output translations given the source or reference text.”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, background): “Since running a comparison of direct assessments (DA, Graham et al., 2013, 2014, 2016) and relative ranking in 2016 (Bojar et al., 2016) and verifying a high correlation of system rankings for the two methods, as well as the advantages of DA, such as quality controlled crowd-sourcing and linear…”
Efficient Online Scalar Annotation with Bounded Support (2018, methodology): “…the annual shared task evaluation of the WMT, DA has been used for scoring adequacy and ﬂuency of machine learning system outputs with human evaluation (Graham et al., 2013, 2014; Bojar et al., 2016, 2017), and has separately been used in creating datasets such as for factuality (Lee et al., 2015).”
Can machine translation systems be evaluated by the crowd alone (2015, methodology): “…(Graham et al. 2013); • It incorporates mechanisms for quality control, based on internal consistency over pairings of original and ‘degraded’ translations (Graham et al. 2014); • It is backwards-compatible with the style of system preference judgment used for WMT evaluations, and provides a…”