Evaluating the morphological competence of Machine Translation Systems

By Franck Burlot, François Yvon. 2017

63 citations recorded by the source corpus.

While recent changes in Machine Translation state-of-the-art brought translation quality a step further, it is regularly acknowledged that the standard automatic metrics do not provide enough insights to fully measure the impact of neural models. This paper proposes a new type of evaluation focused specifically on the morphological competence of a system with respect to various grammatical phenomena. Our approach uses automatically generated pairs of source sentences, where each pair tests one morphological contrast. This methodology is used to compare several systems submitted at WMT'17 for English into Czech and Latvian.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “This framework was extended to evaluate other properties, but often requiring more sophisticated generation methods like using morphological analyzers/generators (Burlot and Yvon, 2017) or more manual involvement in generation (Bawden et al.”
Stress Test Evaluation for Natural Language Inference (2018, neutral): “This is analogous to recent work (Jia and Liang, 2017; Burlot and Yvon, 2017) on developing automated adversarial evaluation schemes for reading comprehension and machine translation.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “…exist for various phenomena, including ones with “minimal” edits similar to our contrast sets, e.g., in image captioning (Shekhar et al., 2017), machine translation (Sen-nrich, 2017; Burlot and Yvon, 2017; Burlot et al., 2018), and language modeling (Marvin and Linzen, 2018; Warstadt et al., 2019).”
On Measuring Gender Bias in Translation of Gender-neutral Pronouns (2019, neutral): “The corpus generation scheme can be compared with Burlot and Yvon (2017) in the sense that various morpho-syntactic/semantic features are taken”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “examples programmatically to study specific phenomena, such as morphology [50], syntax [301], or word sense disambiguation [285].”
Evaluating the Morphosyntactic Well-formedness of Generated Texts (2021, background): “In contrast, there has recently been a burgeoning of evaluation techniques based on grammatical acceptability judgments for both language models (Marvin and Linzen, 2018; Warstadt et al., 2019; Gauthier et al., 2020) and MT systems (Sennrich, 2017; Burlot and Yvon, 2017; Burlot et al., 2018).”
The University of Edinburgh’s Neural MT Systems for WMT17 (2017, methodology): “An independent analysis of our EN → CS and EN → LV news systems’ performance with regard to morphology can be found in Burlot and Yvon (2017).”
Adversarial attacks against Fact Extraction and VERification (2019, methodology): “Manual Construction Small adversarial datasets have been manually constructed and successfully used to identify limitations in Machine Translation [3, 11], Sentiment Analysis [15, 25] and Natural Language Understanding [14] systems.”
Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection (2019, background): “Burlot and Yvon (2017) presented a new protocol for evaluating the morphological competence of MT systems, indicating that current translation systems only manage to capture some morphological phenomena correctly. Regarding the application of constraints in NMT, Sennrich et al. (2016) presented a me”
LIUM Machine Translation Systems for WMT17 News Translation Task (2017, unknown): “Other analysis has been done (Burlot and Yvon, 2017) about morphology strength showing good results in EN-LV task.”
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021, background): “, 2018) question answering (Jia and Liang, 2017), machine translation (Burlot and Yvon, 2017) and language model (Mar-”
On the Linguistic Representational Power of Neural Machine Translation Models (2019, background): “esented in so-called challenge sets. Prominent examples include older work on MT evaluation (King and Falkedal1990), as well as more recent evaluations via contrastive translation pairs (Sennrich2017;Burlot and Yvon 2017;Rios Gonzales, Mascarell, and Sennrich2017;Bawden et al.2018). The latter line ”