A STUDY OF TRANSLATION ERROR RATE WITH TARGETED HUMAN ANNOTATION

By Matthew G. Snover, Bonnie J. Dorr, Richard M. Schwartz, Linnea Micciulla, Ralph M. Weischedel. 2005

1,657 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2016 Conference on Machine Translation (2016, methodology): “Differently from the first edition of the task, in which this distance was only measured in terms of Translation Error Rate (TER) (Snover et al., 2006), this year the BLEU (Papineni et al.”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, positive): “This distance is measured in terms of Translation Error Rate (TER) (Snover et al., 2006a), an evaluation metric commonly used in MT-related tasks (e.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “We find that by updating only 0.1% task-specific parameters,7 prefix-tuning is effective in table-to-text generation, outperforming other lightweight baselines (ADAPTER and FT-TOP2) even by updating 30x fewer parameters and achieving a comparable performance with (full) fine-tuning.”
Can Neural Machine Translation be Improved with User Feedback? (2018, methodology): “, 2007), for TER computation the tercom tool (Snover et al., 2006).”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “We utilize the ofﬁcial benchmarking scripts provided which report BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) and TER (Snover et al., 2005).”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “, 2002), METEOR (Denkowski and Lavie, 2014), and TER (Snover et al., 2005) which are also used in the official WebNLG challenge.”
The WebNLG Challenge: Generating Text from RDF Data (2017, neutral): “• TER5 (Snover et al., 2006).”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) and PARENT (Dhingra et al., 2019”
BERTScore: Evaluating Text Generation with BERT (2019, background): “TER (Snover et al., 2006) normalizes edit distance by the number of reference words, and ITER (Panja & Naskar, 2018) adds stem matching and better normalization.”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “cusedonnewstranslationformorethanadecade. Deﬁning and measuring human quality in translation is challenging for a number of reasons. Traditional metrics of translation quality, such as BLEU [29], TER [34] and Meteor [10] measuretranslationqualitybycomparisonwithoneormorehumanreferencetranslations. H”
METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments (2007, background): “Meteor , as well as several other proposed metrics such as GTM (Melamed et al., 2003), TER (Snover et al., 2006) and CDER (Leusch et al., 2006) aim to address some of these weaknesses.”
chrF: character n-gram F-score for automatic MT evaluation (2015, methodology): “Recent investigations have shown that character level n -grams play an important role for automatic evaluation as a part of more complex metrics such as MT E R ATER (Parton et al., 2011) and BEER (Stanojevi´c and Sima’an, 2014a; Stanojevi´c and Sima’an, 2014b).”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “, 2002) and TER (Snover et al., 2006).”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “As an alternate human metric, we also asked workers to postedit the system’s summary to improve its quality, similar to the post-editing step in MT evaluations (Snover et al., 2006).”
Comparing Automatic Evaluation Measures for Image Description (2014, background): “TER measures the number of modifications a human would need to make to transform a candidate Y into a reference X (Snover et al., 2006).”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, neutral): “TER (Snover et al., 2006) and BEER (Stanojević and Sima’an, 2014) metrics, even though omitted due to space considerations, are consistent with the reported results.”
(Meta-) Evaluation of Machine Translation (2007, neutral): “• Translation Error Rate (Snover et al., 2006)—”