Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation

By Samuel Läubli, Rico Sennrich, Martin Volk. 2018

295 citations recorded by the source corpus.

Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese–English news translation task. We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents. In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. Our findings emphasise the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Scalable Cross Lingual Pivots to Model Pronoun Gender for Translation (2020, background): “Since standard architectures are sentencebased, they can misrepresent pronoun gender since cues for this are often extra-sentential (Läubli et al., 2018).”
Detecting Hallucinated Content in Conditional Neural Sequence Generation (2020, background): “…such as data-to-text generation (Puduppully et al., 2019), machine translation (MT; Vaswani et al. (2017); Wu et al. (2016)) and text summarization (Rothe et al., 2020) can often generate ﬂuent text that is sometimes preferred to human-written content (L¨aubli et al., 2018; Brown et al., 2020).”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, background): “Recent papers have argued that poor human evaluation practices have led to misleading results, including erroneous claims that MT has achieved human parity (Toral, 2020; Läubli et al., 2018).”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “…work, we would like to extend Prism to paragraph- or document-level evaluation by training a paragraph- or document-level multilingual NMT system, as there is growing evidence that MT evaluation would be better conducted at the document level, rather than the sentence level (Läubli et al., 2018).”
The Fallacy of AI Functionality (2022, result): “However, the study used to make this claim (still prominently displayed in press release materials) was quickly debunked by a series of outside researchers who found that at the document-level, when provided with context from nearby sentences, and/or compared to human experts, the machine translatio”
Big Bidirectional Insertion Representations for Documents (2019, background): “Recent research suggests we are nearing human-level parity for sentence-level translation in certain domains (Hassan et al., 2018), however, we lag significantly behind in document-level translation (Lubli et al., 2018).”
Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals (2020, methodology): “4; Methods 13), which was proposed to be a potential confounding factor by the WMT organizers 17 and others 22,23 .”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, background): “We direct the reader to (Läubli et al., 2018) and (Toral increasingly important.”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, methodology): “We follow the protocol proposed by (Läubli et al., 2018, 2020).”
GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, background): “The discourse phenomena, such as pronominal anaphora and lexical consistency will be neglected by sentence-level MT systems, which makes it far worse than human translations [21, 22].”
Neural Machine Translation: A Review (2019, background): “Several studies on document-level NMT indicate that automatic and human sentence-level evaluation metrics often do not correlate well with improvements in discourse level phenomena [527, 594, 607].”
Towards Understanding Neural Machine Translation with Word Importance (2019, background): “Several researchers turn to expose systematic differences between human and NMT translations (Läubli et al., 2018; Schwarzenberg et al., 2019), indicating the linguistic properties worthy of investigating.”
When Combating Hype, Proceed with Caution (2021, background): “…of human parity in machine translation performance (Hassan et al., 2018) severely overstates what has been accomplished relative to commonsense intuitions about what a human-level translation system would do (Toral et al., 2018; Läubli et al., 2018; Zhang and Toral, 2019; Graham et al., 2020).”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, background): “A simple explanation that should nevertheless be empirically validated is that the wording of the sentence created for and within the context of the document does not sound flawless and natural when evaluated in isolation (Läubli et al., 2018a; Toral et al., 2018b).”