Results of the WMT14 Metrics Shared Task

By Matous Machácek, Ondrej Bojar. 2014

141 citations recorded by the source corpus.

This paper presents the results of the WMT14 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in WMT14 Shared Translation Task. We collected scores of 23 metrics from 12 research groups. In addition to that we computed scores of 6 standard metrics (BLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT14 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “As part of these shared tasks, Macháček and Bojar (2014) and Stanojević et al.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “We test two tasks (1) language modeling on part of the RoBERTa corpus including Books (Zhu et al., 2015), CC-News (Nagel, 2016), OpenWebText (Gokaslan and Cohen, 2019), and CC-Stories (Trinh and Le, 2018); and (2) neural machine translation (NMT) (Ott et al., 2018) on WMT14+WMT16 (Macháˇcek and Boja”
SPICE: Semantic Propositional Image Caption Evaluation (2016, methodology): “Our choice of correlation coeﬃcients is consistent with an emerging consensus from the WMT Metrics Shared Task [33,34] for scoring machine translation metrics.”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task (Callison-Burch et al., 2011; Machácek and Bojar, 2014; Stanojevic et al., 2015).”
Results of the WMT15 Metrics Shared Task (2015, methodology): “For a detailed discussion on other options, see Mach´aˇcek and Bojar (2014).”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, methodology): “Other metrics include SentBLEU, NIST, chrF, TER, WER, PER, CDER, and METEOR (Lavie and Agarwal, 2007) that are used and described in the WMT metrics shared task (Bojar et al., 2017; Ma et al., 2018).”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “In the rare cases where significance tests have been applied, tests or confidence intervals for individual correlations form the basis for drawing conclusions (Aziz et al., 2012; Machacek and Bojar, 2014).”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “…is defined by the following matrix: Metric s1 < s2 s1 = s2 s1 > s2 H um an s1 < s2 Conc Disc Disc s1 = s2 − − − s1 > s2 Disc Disc Conc In the notation of Macháček and Bojar (2014), this corresponds to the setup used in WMT12 (with a different underlying method of manual judgements, RR): Metric…”
Results of the WMT17 Metrics Shared Task (2017, background): “For a detailed discussion on other options, see Mach´aˇcek and Bojar (2014).”
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU (2014, methodology): “Following WMT 2013’s metric task (Macháček and Bojar, 2013), for the segment level, we use Kendall’s rank correlation coefficient τ to measure the correlation with human judgment:”
YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources (2019, background): “BLEU (Papineni et al., 2002) has long been shown not to correlate well with human judgment on translation quality (Machacek and Bojar, 2014; Stanojevi´c et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018).”
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2020, unknown): “…NLP tasks such as language understanding (Wang et al., 2018), machine reading comprehension (Rajpurkar et al., 2016, 2018), machine translation (Machacek and Bojar, 2014), summarization (Narayan et al., 2018), etc. ; ( ii ) many researchers are left behind as they do not have expensive GPUs…”