SMART: Sentences as Basic Units for Text Evaluation

By Reinald Kim Amplayo, Peter J. Liu, Yao Zhao, Shashi Narayan. 2022

27 citations recorded by the source corpus.

Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality. In this paper, we introduce a new metric called SMART to mitigate such limitations. Specifically, We treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences. Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation. Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current model-based metrics. The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required. Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards specific models.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “These include accuracy of parse trees (Bangalore et al., 2000), overlap between “basic elements” (Hovy et al., 2005),5 automatically constructed content units (Tauchmann & Mieskes, 2020) using the Pyramid framework by Nenkova and Passonneau (2004), dependency parses (Pratapa et al.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “Therefore, evaluating the claim against individual context sentences (as in previous sentence level evaluation; Laban et al., 2022; Amplayo et al., 2022) can degrade metric performance as paragraph- and documentlevel semantic information is lost.”