Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluation

By Nicholas Egan, Oleg V. Vasilyev, John Bohannon. 2021

22 citations recorded by the source corpus.

The goal of a summary is to concisely state the most important information in a document. With this principle in mind, we introduce new reference-free summary evaluation metrics that use a pretrained language model to estimate the information content shared between a document and its summary. These metrics are a modern take on the Shannon Game, a method for summary quality scoring proposed decades ago, where we replace human annotators with language models. We also view these metrics as an extension of BLANC, a recently proposed approach to summary quality measurement based on the performance of a language model with and without the help of a summary. Using transformer based language models, we empirically verify that our metrics achieve state-of-the-art correlation with human judgement of the summary quality dimensions of both coherence and relevance, as well as competitive correlation with human judgement of consistency and fluency.

Subfields: Artificial Intelligence, Natural Language Processing, Summarization

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “A similar approach based on information theory estimates the extent to which a generated summary helps reconstruct the article on which the summary is based (Egan et al., 2022).”
Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation (2021, methodology): “Egan et al. (2021) proposed automatic Shannon Game (Hovy and Lin, 1998) to measure the decrease of the information one can gain from a document after observing its summary; Peyrard (2019) conducted a theoretical analysis to characterize the information change among source document, background…”