Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems

By Michael J. Denkowski, Alon Lavie. 2011

514 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The NarrativeQA Reading Comprehension Challenge (2017, methodology): “We evaluate using Bleu-1, Bleu-4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011), and Rouge-L (Lin, 2004), using two references for each question, 6 except for the human baseline where we evaluate one reference against the other.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “Trained models are evaluated with respect to BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2011) using MULTEVAL (Clark et al., 2011) and BEER (Stanojević and Sima’an, 2014) to cover a diverse set of automatic measures for translation quality.6 We test for statistical significance with…”
Meteor Universal: Language Specific Translation Evaluation for Any Target Language (2014, methodology): “This constitutes the full Meteor paraphrasing pipeline that has been used to build tables for fully supported languages (Denkowski and Lavie, 2011).”
Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Traditional metrics of translation quality, such as BLEU [29], TER [34] and Meteor [10] measure translation quality by comparison with one or more human reference translations.”
TreeTalk: Composition and Compression of Trees for Image Descriptions (2014, methodology): “, 2002)8 and METEOR (Denkowski and Lavie, 2011).”
Comparing Automatic Evaluation Measures for Image Description (2014, background): “This could be caused by the smaller sample size or because the descriptions were generated by a computer, and not retrieved from a collection of human-written descriptions containing the gold-standard text, as in the Flickr8K data set.”
Accurate Evaluation of Segment-level Machine Translation Metrics (2015, background): “Our large-scale human evaluation reveals three metrics to not be signiﬁcantly outper-formed by any other metric in both Spanish-to-1190 English and a combined evaluation across nine language pairs, namely: METEOR , NLEPOR and SENT BLEU-MOSES .”
Politeness Transfer: A Tag and Generate Approach (2020, methodology): “We also report ROUGE (ROU) (Lin, 2004) and METEOR ( MET ) (Denkowski and Lavie, 2011) scores.”
TWEETQA: A Social Media Focused Question Answering Dataset (2019, methodology): “Thus we consider the task of answer generation for T WEET QA and we use several standard metrics for natural language generation to evaluate QA systems on our dataset, namely we consider BLEU-1 5 (Papineni et al., 2002), Me-teor (Denkowski and Lavie, 2011) and Rouge-L (Lin, 2004) in this paper.”
COMET: A Neural Framework for MT Evaluation (2020, background): “…al., 2002), M ETEOR (Lavie and Denkowski, 2009), and CHR F (Popovi´c, 2015) have been widely studied and improved (Koehn et al., 2007; Popovi´c, 2017; Denkowski and Lavie, 2011; Guo and Hu, 2019), but, by design, they usually fail to recognize and capture semantic similarity beyond the lexical…”
Towards Content Transfer through Grounded Text Generation (2019, methodology): “Our primary automated evaluation metric for system-generated update sentences is ROUGE-L F1 against reference update sentence,7 though we also include BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2011) as additional indicators.”
Dealing with Input Noise in Statistical Machine Translation (2012, methodology): “In case of METEOR we computed the oracle results considering the best hypothesis from the obtained N-best list (1000-best for dev and 50-best for test).”
The First Surface Realisation Shared Task: Overview and Evaluation Results (2011, background): “TER and METEOR were less sensitive, with STUMABA-S and DCU falling into a top group for TER on the test section (i.e., there was no significant difference between STUMABA-S and DCU on the mean TER score at the 0.05 level overall), and STUMABA-S, DCU and STUMABA-D forming a top group for METEOR.”
A Human Judgement Corpus and a Metric for Arabic MT Evaluation (2014, methodology): “We did not observe a strong correlation with the Arabic-tuned ME-TEOR.”
Machine Translation without Words through Substring Alignment (2012, methodology): “These are given partial credit by character-based BLEU (and to a lesser extent METEOR), but marked entirely wrong by word-based BLEU.”
Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation (2015, methodology): “Another obvious extension of the relational dependency LM is the inclusion of more context, for instance through larger windows for siblings and ancestors, or source-context as in (Devlin et al., 2014).”
UvA-DARE ( Digital Academic Repository ) Latent Domain Translation Models in Mix-of-Domains Haystack (2014, unknown): “We report BLEU (Papineni et al., 2002), METEOR 1.4 (Denkowski and Lavie, 2011) and TER (Snover et al., 2006).”