Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts

By Elizabeth Clark, Asli Celikyilmaz, Noah A. Smith. 2019

183 citations recorded by the source corpus.

For evaluating machine-generated texts, automatic methods hold the promise of avoiding collection of human judgments, which can be expensive and time-consuming. The most common automatic metrics, like BLEU and ROUGE, depend on exact word matching, an inflexible approach for measuring semantic similarity. We introduce methods based on sentence mover’s similarity; our automatic metrics evaluate text in a continuous space using word and sentence embeddings. We find that sentence-based metrics correlate with human judgments significantly better than ROUGE, both on machine-generated summaries (average length of 3.4 sentences) and human-authored essays (average length of 7.5). We also show that sentence mover’s similarity can be used as a reward when learning a generation model via reinforcement learning; we present both automatic and human evaluations of summaries learned in this way, finding that our approach outperforms ROUGE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…distance (Kusner et al., 2015), which relies on non-contextual word embeddings, and BERT-SCORE (Zhang et al., 2020), which aggregates cosine distances between represented tokens in a sequence, among others (Zhao et al., 2019; Clark et al., 2019; Kane et al., 2020; Colombo et al., 2021, inter alia).”
BERTScore: Evaluating Text Generation with BERT (2019, background): “, 2019), and SMS (Clark et al., 2019) propose to use optimal matching based on earth mover’s distance (Rubner et al.”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “More recently, Clark et al. (2019) present a semantic metric relying on sentence mover’s similarity and the ELMo representations (Peters et al., 2018) and apply them to summarization and essay scoring.”
Evaluation of Text Generation: A Survey (2020, methodology): “Sentence Mover’s Distance (SMD) Sentence Mover’s Distance (SMD) is an automatic metric based on WMD to evaluate text in a continuous space using sentence embeddings (Clark et al., 2019; Zhao et al., 2019).”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, background): “…for measuring quality of generated text fail to capture structural aspects of language like negation and poorly correlate with human judgements (Hashimoto et al., 2019; Clark et al., 2019; Sellam et al., 2020), leading to a rapidly progressing search for factuality-driven summarization metrics.”
Intrinsic Evaluation of Summarization Datasets (2020, background): “…the shortcomings of ROUGE /automated evaluation (Liu and Liu, 2008; Chaganty et al., 2018; Hashimoto et al., 2019; Peyrard, 2019b) as well proposed alternative metrics for summarization or natural language generation more broadly (Clark et al., 2019; Zhang et al., 2020; Sellam et al., 2020).”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “More recent reference-based metrics are capable of comparisons in a high dimensional space [53, 63, 51, 9], thereby capturing distributional semantics beyond superficial n-gram statistics.”
Re-evaluating Evaluation in Text Summarization (2020, background): “Sentence Mover Similarity (SMS) applies minimum distance matching between text based on sentence embeddings (Clark et al., 2019).”
Counterfactual Story Reasoning and Generation (2019, background): “More recently, Clark et al. (2019) proposed Sentence + Word Mover’s Similarity (S+WMS) to extend WMS for longer multi-sentence texts by using sentence representations in the minimum distance calculation in addition to word embeddings.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “The function g(r, p) measures the similarity between the proposed response p and reference response r using sentence mover’s similarity [24] with Elmo embeddings [123].”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, methodology): “For instance, the Sentence Mover’s Similarity (Clark et al., 2019) represents sentences as the average of their ELMo word embeddings (Peters et al., 2018) and measures the minimum cost of transforming one summary into the other, using a modified version of the Word Mover’s Distance (Kusner et al., 2”
Text Generation by Learning from Demonstrations (2020, background): “Second, while exploration may lead to high-quality sequences different from the given reference, we lack a good reward function to identify these sequences at the moment (Novikova et al., 2017; Aharoni & Goldberg, 2018; Clark et al., 2019).”
NUBIA: NeUral Based Interchangeability Assessor for Text Generation (2020, methodology): “Sentence Mover’s Similarity (Clark et al., 2019) uses a Wasserstein metric defined on sentence embeddings generated from averaging the word embeddings in a sentence.”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, background): “…exact n-gram matching is too rigid, METEOR (Baner-jee and Lavie, 2005; Denkowski and Lavie, 2014) provides ﬂexibility by stemming, synonyms, etc., and recently, a few metrics enable “soft” matching through contextualized word embeddings (Zhao et al., 2019; Clark et al., 2019; Zhang et al., 2019).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “However, recent works have proposed metrics that make a large step toward reliable automatic evaluation of generated text [227], [228].”
SMART: Sentences as Basic Units for Text Evaluation (2022, methodology): “Note that, unlike most of previous work on summarization evaluation (Lin, 2004; Clark et al., 2019; Bhandari et al., 2020; Fab-bri et al., 2021), we also take into account source documents S when calculating metric f q ( · ) .”
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023, background): “( Clark et al., 2019) propose a metric that evaluates multi-sentence texts by computing the similarity between the generated text and the reference text based on the sentence embeddings.”
Measuring Similarity of Opinion-bearing Sentences (2021, methodology): “Embedding-based Metrics Embedding-based metrics are proposed to overcome the limitations of lexical-based metrics (Zhelezniak et al., 2019; Clark et al., 2019; Zhang et al., 2020).”