Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization

By Gabriel Murray, S. Renals, J. Carletta, Johanna D. Moore. 2005

565 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “We train Supervised+RL with a METEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, methodology): “The evaluation metrics include BLEU [61], METEOR [5], ROUGE[49], CIDEr [80] and SPICE [2].”
A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version (2015, positive): “One approach is to borrow evaluation metrics from other NLP tasks such as machine translation, which uses BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) scores.”
The Design and Implementation of XiaoIce, an Empathetic Social Chatbot (2018, background): “Systems BLEU-4 ROUGE-L CIDEr-D METEOR SPICE LSTM-XE Vinyals et al. [2015] 2.96 11.6 1.74 10.43 3.27 LSTM-RL Rennie et al. [2017] 3.43 12.3 2.08 11.84 3.64 DMSM Fang et al. [2015] 2.73 10.52 1.22 11.37 2.63 Up-Down Anderson et al. [2018] 3.23 12.73 1.52 12.66 2.69 XiaoIce (prototype) 4.53 15.33 3.21 ”
Abductive Commonsense Reasoning (2019, methodology): “Among automatic metrics, we report BLEU4 (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), ROUGE (Lin, 2004), CIDEr (Vedantam et al., 2015) and BERT-Score (Zhang et al., 2019) (with the bert-base-uncased model).”
The IIT Bombay English-Hindi Parallel Corpus (2017, methodology): “We evaluated our system using BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005).”
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language (2014, methodology): “The METEOR metric (Banerjee & Lavie, 2005) was designed to resolve various weaknesses of the BLEU and NIST metrics, and it is more focused on word-to-word matches between the reference Algorithm 5 WASPER-GEN-IGSL input sentences S and their associated sets of meaning representations MR(s) output…”
Challenges in Building Intelligent Open-domain Dialog Systems (2019, background): “However, [46] argued that automatic metrics such as BLEU, ROUGE [43], and METEOR [4] all have low correlation with manual evaluation.”
Language Models are General-Purpose Interfaces (2022, methodology): “, 2015), METEOR (Banerjee and Lavie, 2005), and SPICE (Anderson et al.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Our reasons for opting for ranking rather than generation are two fold: (i) Metrics for evaluating language generation (e.g. BLEU [62], METEOR [8]) are focused on local matches (n-grams, longest matching sequences, etc).”
Guiding Extractive Summarization with Question-Answering Rewards (2019, neutral): “The reward function R ( y ) successfully combines intrinsic measures of summary ﬂuency and adequacy (Goldstein et al., 2005) with extrinsic measure of summary responsiveness to given questions (Dang, 2006; Murray et al., 2008).”
HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models (2019, background): “These tasks often resort to automatic metrics like the Inception Score (IS) [52] and Fréchet In-ception Distance (FID) [23] to evaluate images and BLEU [43], CIDEr [56] and METEOR [2] scores to evaluate text.”
Topic-based Evaluation for Conversational Bots (2018, methodology): “Dialog researchers have used metrics of word overlap between a bot response and a reference valid response [19, 10, 23, 24], such as BLEU [17], METEOR [2], and embedding based metrics [24].”
Unifying Multimodal Transformer for Bi-directional Image and Text Generation (2021, methodology): “We report five commonly-used auto-maticevaluationmetricsfortheimagecaptioning task:BLEU@N[33] (N=1,4), ROUGE-L [28], MEREOR [4], CIDEr-D [46] and SPICE [1], which are denoted by B@N, M, R, C, and S for abbreviation.”