METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

By Satanjeev Banerjee, Alon Lavie. 2005

8,297 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Most existing metrics are not consistent (Banerjee and Lavie, 2005) and for the same model vary based on the domain or even the speciﬁc sentences that they are being evaluated on.”
Findings of the 2015 Workshop on Statistical Machine Translation (2015, positive): “The quality estimation task ( § 4) this year included three subtasks: sentence-level prediction of post-editing effort scores, word-level prediction of good/bad labels, and document-level prediction of Meteor scores.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, positive): “We include multiple “traditional” metrics as baseline metrics, notably BLEU (Papineni et al., 2002), ROUGE-1/2/L (Lin, 2004), and METEOR (Banerjee and Lavie, 2005).”
MLSUM: The Multilingual Summarization Corpus (2020, methodology): “METEOR The Metric for Evaluation of Translation with Explicit ORdering (Banerjee and Lavie, 2005) was designed for the evaluation of machine translation output.”
Augmented Language Models: a Survey (2023, methodology): “, 2002a), METEOR (Banerjee and Lavie, 2005), and chrF++ (Popović, 2017), or text2graph ones such as F1, Precision, and Recall.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, methodology): “5), we report the numerical values returned by the COCO on-line evaluation server ‡ , using the CIDEr (Vedantam et al., 2015), ROUGE-L (Lin and Och, 2004), and METEOR (Banerjee and Lavie, 2005) metrics.”
Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “Text Overlap Metrics, including BLEU (Papineni et al. 2002), METEOR (Banerjee and Lavie 2005) and ROUGE (Lin 2004), are the most popular metrics employed in the evaluation of NLG models.”
Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “These metrics primarily rely on token-level overlap over a corpus (also synonymy in the case of METEOR), and therefore are not well-suited for dialog generation since a valid conversational response may not have any token-level or even semantic-level overlap with the ground truths.”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “We train Supervised+RL with a METEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model.”
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations (2021, methodology): “We only report automatic metrics (BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and BERTScore (Zhang et al., 2020)) for NLE quality for ComVE, since WinoGrande has only a small number of test NLE instances (which have been used for grounding in our human evaluation – see Appendix ”
DART: Open-Domain Structured Data Record to Text Generation (2020, methodology): “To evaluate the performance of different NLG methods, qualitative evaluation measures such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005), are widely used to replace costly human judgments.”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, positive): “Table 3 shows that the LongForm models consistently outperform baseline LMs in all tasks, with LongForm-LLaMA-7B achieving a 35% higher rel-Model ative METEOR score than the best baseline LM, Alpaca-LLaMA-7B.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “We report metrics for BLEU [54], METEOR [4] ROUGE [42], and CIDEr [69].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “Evaluation metrics To measure the performance on image caption generation, we consider the standard metrics BLEU-1,4 [59], ROUGE-L [48], METEOR [10], CIDEr-D [74], and SPICE [4].”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “We adopt the corpus-level generation evaluation metric using open-source tool 4 including BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin and Och, 2004) and CIDEr (Vedantam et al., 2015).”
Attention on Attention for Image Captioning (2019, methodology): “We use different metrics, including BLEU [29], METEOR [33], ROUGE-L [13], CIDErD [36] and SPICE [1], to evaluate the proposed method and compare with other methods.”