MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

By Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, Steffen Eger. 2019

732 citations recorded by the source corpus.

A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…distance (Kusner et al., 2015), which relies on non-contextual word embeddings, and BERT-SCORE (Zhang et al., 2020), which aggregates cosine distances between represented tokens in a sequence, among others (Zhao et al., 2019; Clark et al., 2019; Kane et al., 2020; Colombo et al., 2021, inter alia).”
Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021, methodology): “We use the official evaluation script13 and report BLEU, METEOR, TER, MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2020b) and BLEURT (Sellam et al., 2020).”
DART: Open-Domain Structured Data Record to Text Generation (2020, background): “…2018; Gehrmann et al., 2018; Puduppully et al., 2018, 2019; Iso et al., 2019; Castro Ferreira et al., 2019; Shen et al., 2019; Roberti et al., 2019; Zhu et al., 2019b; Koncel-Kedziorski et al., 2019; Ye et al., 2020; Harkous et al., 2020; Wang et al., 2020b; Song et al., 2020; Shahidi et al.,…”
GPTScore: Evaluate as You Desire (2023, background): “MoverScore (Zhao et al., 2019) considers both contextual representation and Word Mover’s Distance (Kusner et al., 2015).”
Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models (2022, background): “…BLEU (Papineni et al., 2002), NIST (Belz and Reiter, 2006), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), TER (Snover et al., 2005), MoverScore (Zhao et al., 2019), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) and PARENT (Dhingra et al., 2019”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “MOVERSCORE (Zhao et al., 2019) combines contextual embeddings and WMD for text generation evaluation.”
BLEURT: Learning Robust Metrics for Text Generation (2020, methodology): “We run MoverScore on WMT 2017 using the scripts published by the authors.”
Evaluation of Text Generation: A Survey (2020, methodology): “Sentence Mover’s Distance (SMD) Sentence Mover’s Distance (SMD) is an automatic metric based on WMD to evaluate text in a continuous space using sentence embeddings (Clark et al., 2019; Zhao et al., 2019).”
COMET: A Neural Framework for MT Evaluation (2020, background): “Again, we contrast our three C OMET models against baseline metrics such as B LEU and CHR F, the 2019 task winning metric Y I S I -1, as well as the recently published metrics B ERTSCORE and B LEURT .”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “For example, BERTScore [63] and MoverScore [64] take features extracted by BERT [7] and apply unsupervised matching functions to compare system outputs against references.”
SummEval: Re-evaluating Summarization Evaluation (2020, background): “…when used outside of its original setting (Liu and Liu, 2008; Cohan and Goharian, 2016) as well as the introduction of many variations on ROUGE (Zhou et al., 2006; Ng and Abrecht, 2015; Ganesan, 2015; ShafieiBavani et al., 2018) and other text generation metrics (Peyrard, 2019; Zhao et al.,…”
SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization (2020, methodology): “…elicit human-written reference summaries and measure their overlap with the generated summaries (e.g. using ROGUE (Lin, 2004a) or MoverScore (Zhao et al., 2019)), or collect some human annotations (e.g. preferences over pairs of summaries (Gao et al., 2019a)) to learn a summary evaluation…”
Re-evaluating Evaluation in Text Summarization (2020, methodology): “However, due to time required and relatively high cost of annotation, the great majority of research papers on summarization use exclusively automatic evaluation metrics, such as ROUGE (Lin, 2004) , JS-2 (Louis and Nenkova, 2013), S3 (Peyrard et al., 2017), BERTScore (Zhang et al., 2020), Mover-Scor”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “Unsupervised learned metrics attempt to obtain training objectives other than human ratings (Zhang et al., 2019; Zhao et al., 2019; Thompson and Post, 2020; Yuan et al., 2021).”
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021, methodology): “We also evaluate our model on the recently developed semantic similarity metrics, namely, BERTScore (Zhang et al., 2020b) and MoverScore (Zhao et al., 2019).”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, methodology): “, 2020), MoverScore (Zhao et al., 2019), BLEURT (Sellam et al.”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “They can be divided into lexical overlap-based (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005) as well as contextualized embedding-based (Zhang et al., 2019; Zhao et al., 2019; Clark et al., 2019b) evaluators.”
Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries (2021, background): “, 2008), BEwTE (Tratz and Hovy, 2008), MeMoG (Giannakopoulos and Karkaletsis, 2011), METEOR (Denkowski and Lavie, 2014), MoverScore (Zhao et al., 2019), NPowER (Giannakopoulos and Karkaletsis, 2013), PyrEval (Gao et al.”
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary (2020, background): “Recent methods BERTScore (Zhang et al., 2020) and MoverScore (Zhao et al., 2019) instead compare tokens based on the similarity of their contextual word embeddings.”