Translationese as a Language in “Multilingual” NMT

By Parker Riley, Isaac Caswell, Markus Freitag, David Grangier. 2019

52 citations recorded by the source corpus.

Machine translation has an undesirable propensity to produce “translationese” artifacts, which can lead to higher BLEU scores while being liked less by human raters. Motivated by this, we model translationese and original (i.e. natural) text as separate languages in a multilingual model, and pose the question: can we perform zero-shot translation between original source text and original target text? There is no data with original source and original target, so we train a sentence-level classifier to distinguish translationese from original target text, and use this classifier to tag the training data for an NMT model. Using this technique we bias the model to produce more natural outputs at test time, yielding gains in human evaluation scores on both accuracy and fluency. Additionally, we demonstrate that it is possible to bias the model to produce translationese and game the BLEU score, increasing it while decreasing human-rated quality. We analyze these outputs using metrics measuring the degree of translationese, and present an analysis of the volatility of heuristic-based train-data tagging.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Machine Translation, Linguistics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Orhan Firat
2. Ankur Bapna
3. Markus Freitag
4. Felix Stahlberg
5. David Grangier
6. Isaac Caswell
7. Graham Neubig
8. Colin Cherry

In Information Systems

1. Markus Freitag
2. David Grangier
3. Ankur Bapna
4. Colin Cherry
5. Orhan Firat
6. Johnny Tian-Zheng Wei

In Natural Language Processing

1. Orhan Firat
2. Ankur Bapna
3. Markus Freitag
4. Isaac Caswell
5. Philipp Koehn
6. David Grangier
7. Felix Stahlberg
8. Graham Neubig

In Machine Translation

1. David Grangier
2. Markus Freitag
3. Ankur Bapna
4. Orhan Firat
5. Isaac Caswell
6. Colin Cherry

In Linguistics

1. Isaac Caswell

Citations in the corpus

• Scaling Laws for Neural Machine Translation (2021, background): “While machine translation by design has composition bias in the training/test sets employed (Freitag et al., 2020; Riley et al., 2020), its effect on model scaling is unknown.”
• BLEU Might Be Guilty but References Are Not Innocent (2020, background): “The impact of translationese in test sets is related to but different from the impact of translationese in the training data (Kurokawa et al., 2009; Lembersky et al., 2012; Bogoychev and Sennrich, 2019; Riley et al., 2019).”
• Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus (2026, unknown): “However, research from 2018 to 2025 reveals that machine-generated translations still exhibit distinct linguistic footprint known as translationese [24, 31, 56].”