A Study of Translation Edit Rate with Targeted Human Annotation

By Matthew G. Snover, Bonnie J. Dorr, Richard M. Schwartz, Linnea Micciulla, John I. Makhoul. 2006

2,223 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “are annotated with the time required for human correction by post-editing and Human-targeted Translation Error Rate (HTER) (Snover et al., 2006).”
Findings of the 2016 Conference on Machine Translation (2016, methodology): “The monolingual alignments between the MT outputs and their post-edits are computed using different methods based on TER, METEOR (Snover et al., 2006) and Berkeley Aligner (Liang et al.”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, positive): “The binary labels for the datasets were acquired automatically with the TERCOM tool (Snover et al., 2006b).”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “MT quality in the Medical Translation Task is evaluated using automatic evaluation metrics: BLEU (Papineni et al., 2002), TER (Snover et al., 2006), PER (Tillmann et al., 1997), and CDER (Leusch et al., 2006).”
The WebNLG Challenge: Generating Text from RDF Data (2017, neutral): “5) 4 (Denkowski and Lavie, 2014); • TER 5 (Snover et al., 2006).”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “We compare the performance of several popular metrics: B LEU (Papineni et al., 2001), CDER (Leusch et al., 2006), and TER (Snover et al., 2006).”
Why We Need New Evaluation Metrics for NLG (2017, methodology): “6Except for TER whose scale is reversed.”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “Such a loss function can be negative smoothed sentence-level evaluation metrics such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), or METEOR (Lavie and Denkowski, 2009) that have been widely used in machine translation evaluation.”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “As an alternate human metric, we also asked workers to post-edit the system’s summary to improve its quality, similar to the post-editing step in MT evaluations (Snover et al., 2006).”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, neutral): “TER (Snover et al., 2006) and BEER (Stanojević and Sima’an, 2014) metrics, even though omitted due to space considerations, are consistent with the reported results.”
Findings of the 2017 Conference on Machine Translation (WMT17) (2017, methodology): “Table 26 shows, for the same data sets, the Translation Error Rate (TER) (Snover et al., 2006a) and the BLEU score (Papineni et al.”
(Meta-) Evaluation of Machine Translation (2007, background): “• Translation Error Rate (Snover et al., 2006)— (4)The GTM scores presented here are an F-measure with a weight of 0.”
Fast Domain Adaptation for Neural Machine Translation (2016, methodology): “All experiments are evaluated with both B LEU [Papineni et al., 2002] and T ER [Snover et al., 2006] (both are case-sensitive).”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, neutral): “Table 2 reports evaluation results in terms of BLEU evaluated on the benchmark test set from JW300, and human-targeted TER (HTER) (Snover et al., 2006), BLEU (Papineni et al., 2002) and ChrF (Popovi´c, 2015) against human corrected model translations.”
Convolutional Neural Network for Paraphrase Identification (2015, background): “Madnani et al. (2012) treated paraphrase relationship as a kind of mutual translation, hence combined eight kinds of machine translation metrics including BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), TERp (Snover et al., 2009), METEOR (Denkowski and Lavie, 2010),”
Results of the WMT15 Metrics Shared Task (2015, methodology): “The metrics TER (Snover et al., 2006), WER, PER and CDER (Leusch et al., 2006) were computed using the Moses scorer which is used in Moses model optimization.”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “The metrics TER (Snover et al., 2006), WER , PER and CDER (Leusch et al., 2006) were produced by the Moses scorer, which is used in Moses model optimization.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “terp Version of ter handling phrasal substitution, stemming and synonymy (Snover et al., 2006).”