A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines

By Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Jan-Thorsten Peter, P. Williams. 2017

94 citations recorded by the source corpus.

Abstract In this paper, we report an analysis of the strengths and weaknesses of several Machine Translation (MT) engines implementing the three most widely used paradigms. The analysis is based on a manually built test suite that comprises a large range of linguistic phenomena. Two main observations are on the one hand the striking improvement of an commercial online system when turning from a phrase-based to a neural engine and on the other hand that the successful translations of neural MT systems sometimes bear resemblance with the translations of a rule-based MT system.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Evaluating Gender Bias in Machine Translation (2019, background): “These include challenge sets for language-speciﬁc idiosyncrasies (Isabelle et al., 2017), discourse phenomena (Bawden et al., 2018), pronoun translation (M¨uller et al., 2018; Webster et al., 2018), or coreference and multiword expressions (Burchardt et al., 2017).”
Analysis Methods in Neural Language Processing: A Survey (2018, methodology): “In MT evaluation, Burchardt et al. (2017) reported results using a large test suite covering 120 phenomena, partly based on Lehmann et al. (1996).”
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (2021, background): “…evaluation by targeting characteristic discourse-level phenomena have been proposed (Hardmeier et al., 2015; Guillou and Hardmeier, 2016; Burchardt et al., 2017; Isabelle et al., 2017; Rios Gonzales et al., 2017; Müller et al., 2018; Bawden et al., 2018; Voita et al., 2019; Guillou and…”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, background): “Following work in NLP [196] and machine translation [178], a number of such suites have been manually constructed to evaluate neural machine translation performance on a range of linguistic phenomena [23, 49, 161].”
On Integrating Discourse in Machine Translation (2017, methodology): “Both Popović (2017); Burchardt et al. (2017) directly or indirectly touch on the issue of evaluation.”