Using Test Suites in Evaluation of Machine Translation Systems

By M. King, Kirsten Falkedal. 1990

As awareness of the increasing need for translations grows, readiness to consider computerized aids to translation grows with it. Recent years have seen increased funding for research in machine aids to translation, both in the public and the private sector, and potential customers are much in evidence in conferences devoted to work in the area. Activity in the area in its turn stimulate,; an interest in evaluation techniques: sponsors would like to know if their money has been well spent, system developers would like to know how well they fare compared to their rivals, and potential customers need to be able to estimate the wisdom of their proposed investment. indeed, interest in evaluation extends beyond translation aids to natural language processing as a whole, as a consequence of attempts to facilitate storage and retrieval of large anaounts of information. Concrete manifestations of this interest include a workshop on evaluation in Philadelphia in late 1988, and, in the particular field of machine translation, the publication of two books, the first [4] dedicated to the topic, the second [7] containing much discussion of it.

Subfields: Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “, 1996), especially for evaluating MT systems (King and Falkedal, 1990; Isahara, 1995; Koh et al., 2001).”
A Challenge Set Approach to Evaluating Machine Translation (2017, background): “We focus on phenomena that we expect to be more difficult than average, resulting in a particularly challenging MT test suite (King and Falkedal, 1990).”
DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation (2019, background): “While hand-crafted examples and challenge sets provide the means to test particular phenomena (King and Falkedal, 1990; Isabelle et al., 2017), it is also important to observe and evaluate the quality of translation on real, spontaneously produced texts.”
A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines (2017, background): “Apart from several singular attempts (King and Falkedal, 1990; Isahara, 1995; Koh et al., 2001, etc.) broadly-defined test suites have not generally been used in MT research.”
On Integrating Discourse in Machine Translation (2017, background): “In the past there have been examples of unit testing for evaluation of MT quality, in particular (King and Falkedal, 1990) who developed theirs for evaluation of different MT systems before financial outlay.”
Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation (2010, background): “A operates over predefined test suites, i.e., over fixed sets of translation test cases (King and Falkedal, 1990).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, unknown): “Following work in NLP [196] and machine translation [178], a number of such suites have been manually constructed to evaluate neural machine translation performance on a range of linguistic phenomena [23, 49, 161].”
Test Suites for Natural Language Processing (1995, background): “First, as King and Falkedal (ibid) point out, most existing test suites are designed for monolingual applications.”