Meteor Universal: Language Specific Translation Evaluation for Any Target Language

By Michael J. Denkowski, Alon Lavie. 2014

2,071 citations recorded by the source corpus.

This paper describes Meteor Universal, released for the 2014 ACL Workshop on Statistical Machine Translation. Meteor Universal brings language specific evaluation to previously unsupported target languages by (1) automatically extracting linguistic resources (paraphrase tables and function word lists) from the bitext used to train MT systems and (2) using a universal parameter set learned from pooling human judgments of translation quality from several language directions. Meteor Universal is shown to significantly outperform baseline BLEU on two new languages, Russian (WMT13) and Hindi (WMT14).

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “4 We also evaluate with the METEOR metric (Denkowski and Lavie, 2014), both in exact match mode (rewarding only exact matches between words) and full mode (which additionally rewards matching stems, synonyms and para-phrases).”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “QG First, we use three traditional automatic evaluation metrics: BLEU4 (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014), ROUGE-L (Lin, 2004).”
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption (2020, methodology): “tions are evaluated by the caption metrics (BLEU [42], METEOR [14], ROUGE L [35], SPICE [3], and CIDEr [53]).”
Learning Visual N-Grams from Web Data (2016, methodology): “5Our model achieves a METEOR score [11] of 17.”
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics (2013, neutral): “Their findings concurred with ours and they suggested using either Meteor [Denkowski and Lavie, 2014], a modified smoothed BLEU score [Clark et al.”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, neutral): “With the exception of SPICE, all the standard textual evaluation metrics we evaluated (BLEU, CIDEr, METEOR and ROUGE) are ineffective, and – perhaps as a result – existing instruction generators have substantial headroom for improvement.”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “, 2002), METEOR (Denkowski and Lavie, 2014), and TER (Snover et al.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, methodology): “, 2002) , ROUGE (Lin, 2004), NIST (Lin and Och, 2004), METEOR (Denkowski and Lavie, 2014) and CiDER (Vedantam et al.”
The WebNLG Challenge: Generating Text from RDF Data (2017, neutral): “5)4 (Denkowski and Lavie, 2014);”
UFO: A UniFied TransfOrmer for Vision-Language Representation Learning (2021, methodology): “The beam search size is set as 1, and the accuracy is evaluated with BLEU@4 [34], METEOR [9], CIDEr [46], and SPICE [3].”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Similar to Li et al. [19], we validate our results over the COCO dataset using the common metrics BLEU [27], METEOR [10], CIDEr [37] and SPICE [3], and for the nocaps dataset using CIDEr and SPICE.”
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “We adopt four diverse automatic evaluation metrics: BLEU [40], Meteor [17], Rouge-L [35], and CIDEr [56].”
Microsoft COCO Captions: Data Collection and Evaluation Server (2015, methodology): “Using the testing data, our evaluation server evaluates captions output by different approaches using numerous automatic metrics: BLEU [39], METEOR [41], • Xinlei Chen is with Carnegie Mellon University.”
Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, methodology): “Following DALL-Eval [55], we also measure text-image fit through automated captioning evaluation (or captioner evaluation): an image output by the model is captioned with a pretrained VL-T5 model [56] and then the similarity of the input prompt and and the generated caption is assessed via BLEU [57]”
GIT: A Generative Image-to-text Transformer for Vision and Language (2022, methodology): “Our model achieves new SOTA performance 1) on all metrics (BLEU@4 [73], METEOR [13], CIDEr [90], SPICE [3]) with SCST and 2) on CIDEr and BLEU@4 with cross-entropy optimization only.”