The Meteor metric for automatic evaluation of machine translation

By Alon Lavie, Michael J. Denkowski. 2009

385 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards a Better Metric for Evaluating Question Generation Systems (2018, methodology): “In particular, there is a need to closely examine the utility of existing n -gram based similarity metrics such as BLEU (Papineni et al., 2002), METEOR (Lavie and Denkowski, 2009), NIST (Doddington, 2002), etc. which have been adopted for this task.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, background): “…correlation between multiple widely used summary evaluation metrics and human judgement scores, so as to further motivate the need for a stronger reward for RL. Metrics we consider include ROUGE (full length F-score), BLEU (Papineni et al., 2002) and METEOR (Lavie and Denkowski, 2009).”
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems (2015, methodology): “to tackle metrics for which we can make progress. Second, we do not yet have a suitable metric for evaluating performance in the response generation case. One option is to use the BLEU [18] or METEOR [16] scores from machine translation. However, using BLEU to evaluate dialogue systems has been show”
Unifying Human and Statistical Evaluation for Natural Language Generation (2019, methodology): “Automated metrics based on n-gram overlap such as BLEU, METEOR, ROUGE (Papineni et al., 2002; Lavie and Denkowski, 2009; Lin and Rey, 2004) work well for machine translation but do not generalize well to domains with a diverse spectrum of correct responses.”
Minimum Risk Training for Neural Machine Translation (2015, methodology): “Such a loss function can be negative smoothed sentence-level evaluation metrics such as BLEU (Papineni et al., 2002), NIST (Doddington, 2002), TER (Snover et al., 2006), or METEOR (Lavie and Denkowski, 2009) that have been widely used in machine translation evaluation.”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “…human evaluation is often too expensive, there has been a large effort develop- ∗Authors contributed equally. ing automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin and Rey, 2004), METEOR (Lavie and Denkowski, 2009; Denkowski and Lavie, 2014) and CiDER (Vedantam et al., 2015).”
COMET: A Neural Framework for MT Evaluation (2020, background): “, 2002), METEOR (Lavie and Denkowski, 2009), and CHRF (Popović, 2015) have been widely studied and improved (Koehn et al.”
Combining Word-Level and Character-Level Models for Machine Translation Between Closely-Related Languages (2012, methodology): “Still, as line (w2) shows, using the 1-best transliteration lattice as an input to (w1) yields3 consistent improvement over (w1) for four evaluation metrics: BLEU (Papineni et al., 2002), NIST v. 13, TER (Snover et al., 2006) v. 0.7.25, and METEOR (Lavie and Denkowski, 2009) v. 1.3.”
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity (2017, background): “Examples include Word Sense Disam-biguation (Miller et al., 2012), Information Retrieval (Hliaoutakis et al., 2006), Machine Translation (Lavie and Denkowski, 2009), Lexical Sub-stitution (McCarthy and Navigli, 2009), Question Answering (Mohler et al., 2011), Text Summa-rization (Mohammad and Hirst,”
Exploring Content Selection in Summarization of Novel Chapters (2020, methodology): “We present a new approach for aligning chapter sentences with the abstractive summary sentences, incorporating weighting to ROUGE (Lin, 2004) and METEOR (Lavie and Denkowski, 2009) metrics to enable the alignment of salient words between them.”
Survey on evaluation methods for dialogue systems (2019, methodology): “This includes measures such as: Levenshtein distance, METEOR (Lavie and Denkowski 2009), or TF-IDF retrieval models (Charras et al.”
Tuning as Ranking (2011, background): “, 2002), though there are several common alternatives (Lavie and Denkowski, 2009; Melamed et al., 2003; Snover et al., 2006; Chiang et al., 2008a).”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, background): “Popular metrics like ROUGE (Lin, 2004a), BLEU (Papineni et al., 2002) and METEOR (Lavie and Denkowski, 2009) fall into this category.”
Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning (2017, methodology): “, 2002), METEOR (Lavie and Denkowski, 2009) and TER (Snover et al.”
TakeLab: Systems for Measuring Semantic Text Similarity (2012, methodology): “Previous research by Lavie and Denkowski (2009) proposed a similar alignment strategy for machine translation evaluation.”
OSU Multimodal Machine Translation System Report (2017, methodology): “WMT organization provides three different evaluating metrics: BLEU (Papineni et al., 2002), METEOR (Lavie and Denkowski, 2009) and TER (Snover et al., 2006).”