METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments

By Alon Lavie, Abhaya Agarwal. 2007

1,442 citations recorded by the source corpus.

Meteor is an automatic metric for Machine Translation evaluation which has been demonstrated to have high levels of correlation with human judgments of translation quality, significantly outperforming the more commonly used Bleu metric. It is one of several automatic metrics used in this year's shared task within the ACL WMT-07 workshop. This paper recaps the technical details underlying the metric and describes recent improvements in the metric. The latest release includes improved metric parameters and extends the metric to support evaluation of MT output in Spanish, French and German, in addition to English.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Findings of the 2015 Workshop on Statistical Machine Translation (2015, neutral): “, 2006), and the edit distance-based aligners (Snover et al., 2006a; Lavie and Agarwal, 2007).”
Semantic Noise Matters for Neural Natural Language Generation (2019, methodology): “We use freely available word-overlap-based evaluation metrics (WOM) scripts that come with the E2E data (Duˇsek et al., 2019), 9 supporting BLEU (Papineni et al., 2002), NIST (Doddington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007) and CIDEr (Vedantam et al., 2015).”
Neural Generation for Czech: Data and Baselines (2019, methodology): “NIST (Doddington, 2002), ROUGE-L (Lin, 2004), METEOR (Lavie and Agarwal, 2007) and CIDEr (Vedantam et al.”
The E2E Dataset: New Challenges For End-to-End Generation (2017, background): “BLEU (Papineni et al., 2002) 0.6925 NIST (Doddington, 2002) 8.4781 METEOR (Lavie and Agarwal, 2007) 0.4703 ROUGE-L (Lin, 2004) 0.7257 CIDEr (Vedantam et al., 2015) 2.3987 Table 6: TGen results on the development set. cent E2E data-driven systems.”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “, 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “We use the standard automatic evaluation metrics including CIDEr (Anderson et al., 2016), BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and SPICE (Anderson et al., 2016).”
HTLM: Hyper-Text Pre-Training and Prompting of Language Models (2021, methodology): “We utilize the ofﬁcial benchmarking scripts provided which report BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) and TER (Snover et al., 2005).”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…the ofﬁcial evaluation metrics, including BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam…”
DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation (2019, methodology): “Beam search (with beam width 10) dramatically improves BLEU and DIST scores, and marginally improves NIST and METEOR.”
nocaps: novel object captioning at scale (2019, methodology): “These metrics test whether models mention novel objects accurately [41] as well as describe them fluently [20].”
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2020, neutral): “To further validate that, we remove the single-modal learning processes on the text corpus and Model Flickr30k-IR Flickr30k-TR SNLI-VE VQA CoCo Caption R@1 / R@5 / R@10 R@1 / R@5 / R@10 Val test-dev BLUE4 image collections (i.e., “w/o single-modal”) from UNIMO and replace the CMCL with an image-text”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, neutral): “The possible alternatives are BLEU score [30], METEOR [18], CIDEr [38] or other metrics that are widely used in the image captioning task [24].”
Generation and Comprehension of Unambiguous Object Descriptions (2015, methodology): “The generation task is more difﬁcult — we can evaluate a generated description in the same way as an image description, using metrics such as CIDEr [51], BLEU [44] and METEOR [35].”
Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge (2019, neutral): “System name BLEU NIST METEOR ROUGE-L CIDEr ♥TGEN 0.08 0.00 0.03 0.04 0.03 ♥ADAPT 0.05 0.05 0.05 0.09 0.09 ♥CHEN 0.03 −0.03 −0.06 0.01 −0.08 ♣DANGNT 0.06 −0.11 0.02 0.06 0.08 ♣FORGE1 −0.06 −0.13∗ −0.03 0.06 0.05 ♠FORGE3 0.03 0.02 −0.03 −0.02 0.04 ♥GONG 0.08 0.10 0.00 0.08 −0.02 ♥HARV 0.04 0.05 0.02 0”
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (2020, methodology): “METEOR (Lavie and Agarwal, 2007) extends BLEU by using an alignment between the generated text and a reference, as well as using stemming and synonym replacement for more ﬂexible n -gram matching.”
(Meta-) Evaluation of Machine Translation (2007, background): “Its flexible matching was extended to French, Spanish, German and Czech for this workshop (Lavie and Agarwal, 2007).”