Sometimes We Want Ungrammatical Translations

By Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, Adina Williams. 2021

12 citations recorded by the source corpus.

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has focused primarily on improving translation quality, and as a secondary focus, improving robustness to perturbations (e.g. spelling). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness or faithfulness, by focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure their effects on the target side. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which prop-erty is best suited for their use case.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, methodology): “[101] calculate faithfulness by computing similarity scores between perturbed source sentence and target sentence after applying the same perturbation.”
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022, background): “Other work in this space has tried to understand what models know about syntax [24,28,34,49,54,71,83] or the complex interaction between syntactic and semantic categories [38,78,81,82].”