CDER: Efficient MT Evaluation Using Block Movements

By Gregor Leusch, Nicola Ueffing, Hermann Ney. 2006

155 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2014 Workshop on Statistical Machine Translation (2014, methodology): “MT quality in the Medical Translation Task is evaluated using automatic evaluation metrics: BLEU (Papineni et al., 2002), TER (Snover et al., 2006), PER (Tillmann et al., 1997), and CDER (Leusch et al., 2006).”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “We compare the performance of several popular metrics: B LEU (Papineni et al., 2001), CDER (Leusch et al., 2006), and TER (Snover et al., 2006).”
METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments (2007, background): “, 2006) and CDER (Leusch et al., 2006) aim to address some of these weaknesses.”
Results of the WMT15 Metrics Shared Task (2015, methodology): “CDER is the best of the baselines, reaching .94 into English and .81 out of English.”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “The baselines serve in system and segment-level evaluations as customary: BLEU , TER , WER , PER and CDER for system-level only; sentBLEU for segment-level only and chrF for both.”
Results of the WMT16 Metrics Shared Task (2016, methodology): “, 2006), MOSESWER, MOSESPER and MOSECDER (Leusch et al., 2006) were produced by the Moses scorer which is used in Moses model http://www.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “, 2006), WER, PER and CDER (Leusch et al., 2006) were produced by the Moses scorer, which is used in Moses model optimization.”
Results of the WMT17 Metrics Shared Task (2017, methodology): “The metrics TER (Snover et al., 2006), WER, PER and CDER (Leusch et al., 2006) were produced by the Moses scorer, which is used in Moses model optimization.”
Evaluation of Text Generation: A Survey (2020, methodology): “CDER (Leusch et al., 2006) models block ordering as an edit operation.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “, 2002; Lin, 2004; Banerjee and Lavie, 2005; Doddington, 2002; Popović, 2015), edit based (Leusch et al., 2006; Snover et al., 2006; Wang et al., 2016) and embedding based metrics (Chow et al.”
Towards Automatic Error Analysis of Machine Translation Output (2011, methodology): “The Translation Edit Rate ( TER ) (Snover et al. 2006) and the CD ER measure (Leusch, Uefﬁng, and Ney 2006) are based on the edit distance ( WER ) but allow reordering of blocks.”
Improving Statistical Machine Translation Using Word Sense Disambiguation (2007, background): “…the recently proposed Meteor (Banerjee and Lavie, 2005) and four edit-distance style metrics, Word Error Rate (WER), Positionindependent word Error Rate (PER) (Tillmann et al., 1997), CDER, which allows block reordering (Leusch et al., 2006), and Translation Edit Rate (TER) (Snover et al., 2006).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, unknown): “rence, the 4The name is a fusion of ‘SPICE’ and ‘CIDEr’ Manuscript submitted to ACM A Survey of Evaluation Metrics Used for NLG Systems 23 remaining words are counted as insertions or deletions. CDER [75] models block reordering as a unit edit operation to off-set unnecessary costs in shifting words”
The Meteor metric for automatic evaluation of machine translation (2009, methodology): “Meteor, along with other metrics such as GTM (Melamed et al., 2003), TER (Snover et al., 2006) and CDER (Leusch et al., 2006), were developed with the aim of specifically addressing this and other weaknesses identified in Bleu .”
BERT: A Review of Applications in Natural Language Processing and Understanding (2021, methodology): “This can also include methods like PER [77] and CDER [37], taking into account the permutation of text blocks.”