Results of the WMT15 Metrics Shared Task

By Miloš Stanojević, Amir Kamran, Philipp Koehn, Ondrej Bojar. 2015

105 citations recorded by the source corpus.

This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task. We collected scores of 46 metrics from 11 research groups. In addition to that, we computed scores of 7 standard metrics (BLEU, SentBLEU, NIST, WER, PER, TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “In addition to hallucinations, Stevens-Guille et al. (2020) also investigate repetitions and omissions of content as plausible errors.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “To address the issue, the machine translation community has been organizing yearly metrics shared tasks which produce metrics that achieve a high correlation (Stanojević et al., 2015; Bojar et al., 2016, 2017; Ma et al., 2018, 2019; Mathur et al., 2020b).”
Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data (2019, background): “CHRF is our primary evaluation metric for EN → KK, due to the fact that this metric has been shown to correlate better than BLEU with human evaluation when the target language is agglutinative (Stanojevi´c et al., 2015).”
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses (2017, methodology): “Related to our approach is the literature on novel methods for the evaluation of machine translation systems, especially through the WMT evaluation task (Callison-Burch et al., 2011; Machácek and Bojar, 2014; Stanojevic et al., 2015).”
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, methodology): “We used segment-level datasets for toEnglish language pairs from the WMT15 (Stanojević et al., 2015), WMT16 (Bojar et al.”
Balancing Cost and Benefit with Tied-Multi Transformers (2020, background): “…the 36 vanilla models in parallel. chrF (Popović, 2016), since it has been particularity designed for sentence-level translation evaluation and was shown to have relatively high correlation with human judgment of translation quality at sentence level for the English–German pair (Ma et al., 2018).”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, methodology): “…of AMR-to-text systems is typically evaluated against the sentence from which the AMRs was created us-ing standard surface string matching metrics such as B LEU (Papineni et al., 2002) or CHR F(++) (Stanojevi´c et al., 2015; Popovi´c, 2015, 2016; Popov, 2017), as employed in general NLG tasks.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “It uses an SVM regressor with 57 metric scores as features and the DA scores (direct assessment scores on translation quality obtained through human evaluators [55]) from WMT15[149] and WMT16[14] as the gold standard target.”
CharacTer: Translation Edit Rate on Character Level (2016, methodology): “We calculated the CharacTER and CHRF3 scores and cited the other scores from the WMT metric papers (Macháček and Bojar, 2013; Macháček and Bojar, 2014; Stanojević et al., 2015).”
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (2021, methodology): “pr et ra in ed BERTScore D — D — 104 Zhang et al. (2020) BLEURT D D D — 104* Sellam et al.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, unknown): “…deﬁned metrics like accuracy, precision, and recall, but also metrics parameterized by models (Frechet Inception Distance [63], B ERT [111], BLEURT [137]), and metrics which involve humans in the loop, like human evaluations of machine translation (Direct Assessment [144], Relative Ranking [51]).”