How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

By Rico Sennrich. 2016

170 citations recorded by the source corpus.

Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval97, a large-scale data set of 97000 contrastive translation pairs based on the WMT English->German translation task, with errors automatically created with simple rules. We report results for a number of systems, and find that recently introduced character-level NMT systems perform better at transliteration than models with byte-pair encoding (BPE) segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

HateCheck: Functional Tests for Hate Speech Detection Models (2020, methodology): “In its use of contrastive cases for model evaluation, H ATE C HECK builds on a long history of minimally-contrastive pairs in NLP (e.g. Levesque et al., 2012; Sennrich, 2017; Glockner et al., 2018; Warstadt et al., 2020).”
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “An alternative method to study the ability of language models and machine translation models to capture hierarchical information is to test their sensitivity to speciﬁc grammatical errors (Linzen et al., 2016; Gulordava et al., 2018; Tran et al., 2018; Sennrich, 2017; Tang et al., 2018).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “As is now common in NMT, we apply Byte Pair Encoding (BPE) (Sennrich, 2017) to target sentences only, with 40k codes (see Table 1 in the main text for examples of transformed target sentences).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, neutral): “While recent advances have reported near human-level performance on several language pairs using neural approaches (Wu et al., 2016; Hassan et al., 2018), other studies have highlighted several open challenges (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017).”
Targeted Syntactic Evaluation of Language Models (2018, background): “…Nivre et al., 2010; Bender et al., 2011), and minimal pair approaches have been proposed for evaluating image captioning (Shekhar et al., 2017) and machine translation systems (Sennrich, 2017), but no data sets exist that target a range of syntactic constructions for language model evaluation.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “, 2017), machine translation (Sennrich, 2017; Burlot and Yvon, 2017; Burlot et al., 2018), and language modeling (Marvin and Linzen, 2018; Warstadt et al.”
A Challenge Set Approach to Evaluating Machine Translation (2017, methodology): “We preprocessed the data using a BPE model learned from source and target corpora (Sennrich et al., 2016).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “, 2018) as well as to a recent evaluation dataset (Sennrich, 2017), tackling also the attention weights between the encoder and the decoder.”
Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures (2018, methodology): “The subject-verb agreement task is the most popular choice for evaluating the ability to capture long-range dependencies and has been used in many studies (Linzen et al., 2016; Bernardy and Lappin, 2017; Sennrich, 2017; Tran et al., 2018).”
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, background): “However, a general repeated observation from the NMT literature is the disconnect between model score and BLEU score.”
Evaluating NLP Models via Contrast Sets (2020, neutral): “translation (Sennrich, 2017), and language modeling (Marvin and Linzen, 2018; Warstadt et al.”