APE at Scale and Its Implications on MT Evaluation Biases

By Markus Freitag, Isaac Caswell, Scott Roy. 2019

69 citations recorded by the source corpus.

In this work, we train an Automatic Post-Editing (APE) model and use it to reveal biases in standard MT evaluation procedures. The goal of our APE model is to correct typical errors introduced by the translation process, and convert the “translationese” output into natural text. Our APE model is trained entirely on monolingual data that has been round-trip translated through English, to mimic errors that are similar to the ones introduced by NMT. We apply our model to the output of existing NMT systems, and demonstrate that, while the human-judged quality improves in all cases, BLEU scores drop with forward-translated test sets. We verify these results for the WMT18 English to German, WMT15 English to French, and WMT16 English to Romanian tasks. Furthermore, we selectively apply our APE model on the output of the top submissions of the most recent WMT evaluation campaigns. We see quality improvements on all tasks of up to 2.5 BLEU points.

Subfields: Artificial Intelligence, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

Citations in the corpus

Scaling Laws for Neural Machine Translation (2021, background): “[9] Markus Freitag, Isaac Caswell, and Scott Roy.”
Controlling Computation versus Quality for Neural Sequence Models (2020, background): “This might also be explained by the model’s outputs being samples of a smooth marginal distribution that is easier to learn (with much fewer modes), as compared against natural target text (Freitag et al., 2019).”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, background): “The impact of translationese (Koppel and Ordan, 2011) on human evaluation of MT has recently received attention (Toral et al., 2018; Zhang and Toral, 2019; Freitag et al., 2019; Graham et al., 2020).”
BLEU Might Be Guilty but References Are Not Innocent (2020, methodology): “Freitag et al. (2019); Edunov et al. (2019) point at translationese as a major source of mismatch between BLEU and human evaluation.”
Translationese as a Language in “Multilingual” NMT (2019, methodology): “Prior work has shown that BLEU has a low correlation with human judgments when the reference contains translationese but the system output is biased toward original/natural text (Freitag et al., 2019).”
Can Automatic Post-Editing Improve NMT? (2020, methodology): “Using a similar approach of generating artificial APE data, Freitag et al. (2019) trained a monolingual re-writing APE model trained on the generated artificial training data alone.”
Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “While the first two strategies haven’t been explored in the context of conditional computation, distillation (Hinton et al., 2015; Kim and Rush, 2016) has been found to introduce undesirable artifacts into the student model (Freitag et al., 2019; Bogoychev and Sennrich, 2019) in the context of NMT.”
The statistical advantage of automatic NLG metrics at the system level (2021, background): “Several studies have shown scoring differences between professional linguists and crowdworkers which are due in part to the fact that linguists are more sensitive to linguistic phenomena (Fabbri et al., 2019; Freitag et al., 2019).”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “Since French was not included in WMT21, we use data from WMT14; apart from being older, these test sets are not purely source-original (Freitag et al., 2019) like the more recent ones.”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, background): “Interestingly, the human evaluation results show no correlation with the automatic scores from Table 46 which confirms the findings from Freitag et al. (2019) that automatic evaluation is hard for post-edited systems.”