Evaluation of machine translation and its evaluation

By Joseph P. Turian, Luke Shea, I. Dan Melamed. Machine Translation Summit, 2003

335 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/b038147589432947cd47b2c75d46e43613a1a91b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2020, background): “In machine translation evaluation for instance, inter-rater agreements have traditionally been reported as low (Turian et al., 2003; Carl et al., 2011; Lommel et al., 2014), especially when quality estimates are collected from non-professional raters (Callison-Burch, 2009).”
Comparing Rating Scales and Preference Judgements in Language Evaluation (2010, background): “It is not surprising then that stable averages of quality judgements, let alone high levels of agreement, are hard to achieve, as has been observed for MT (Turian et al., 2003; Lin and Och, 2004), text summarisation (Trang Dang, 2006), and language generation (Belz and Reiter, 2006).”
Overview of the IWSLT 2011 evaluation campaign (2011, methodology): “Besides the BLEU metric scores, automatic evaluation using six additional standard metrics (METEOR [2], WER [3], PER [4], TER [5], GTM [6], and NIST [7]) were calculated offline and are listed in Appendix A.”
Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics (2004, methodology): “Recently, Turian et al. (2003) indicated that standard accuracy measures such as recall, precision, and the F-measure can also be used in evaluation of machine translation.”
The IWSLT 2011 Evaluation Campaign on Automatic Talk Translation (2012, methodology): “…numbers six additional standard metrics—METEOR (Lavie and Agarwal, 2007), WER (Niessen et al., 2000), PER (Och, 2003), TER (Snover et al., 2006), GTM (Turian et al., 2003), and NIST (Dod-dington, 2002)—were calculated ofﬂine and reported in the evaluation overview paper (Federico et al., 2011).”
A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION (2005, background): “Future studies might benefit from also examining correlations with the MMS evaluation metric(Turian et al., 2003).”
A Study of Translation Edit Rate with Targeted Human Annotation (2006, background): “An MT scoring measure that uses the notion of maximum matching string (MMS) has been demonstrated to yield high correlations with human judges (Turian et al., 2003).”
Best practices for the human evaluation of automatically generated text (2019, result): “However, in contrast to what this theory would predict, both Doddington (2002) and Turian et al. (2003) report that correlations between metrics and human judgments in machine translation do not improve substantially as the number of reference texts increases.”
Paraphrase Identification as Probabilistic Quasi-Synchronous Recognition (2009, background): “…recourse to any hidden structure: proportion of word unigram matches, proportion of lemma-tized unigram matches, BLEU score (Papineni et al., 2001), BLEU score on lemmatized tokens, F measure (Turian et al., 2003), difference of sentence length, and proportion of dependency relation overlap.”
ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation (2004, neutral): “Turian et al. (2003) introduced General Text Matcher (GTM) based on accuracy measures such as recall, precision, and F-measure.”
FFR v1.1: Fon-French Neural Machine Translation (2020, background): “Researchers have shown that automatic metrics are not necessarily a good substitute for human assessments of translation quality (Turian et al., 2003; Callison-Burch et al., 2006; Graham et al., 2016), due to issues like lexical-vs-semantic similarity and existence of many possible valid…”
Mining the Correlation between Human and Automatic Evaluation at Sentence Level (2010, background): “2001), Precision and Recall (Turian et al. 2003), TER (Translation Error Rate) (Snover et al.”
The repetition rate of text as a predictor of the effectiveness of machine translation adaptation (2014, methodology): “First of all, Table 3 provides BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and GTM (Turian et al., 2003) scores computed on the evaluation documents with respect to human references for each of the ﬁve considered translation tasks.”