Finding a Balanced Degree of Automation for Summary Evaluation

By Shiyue Zhang, Mohit Bansal. 2021

55 citations recorded by the source corpus.

Human evaluation for summarization tasks is reliable but brings in issues of reproducibility and high costs. Automatic metrics are cheap and reproducible but sometimes poorly correlated with human judgment. In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but replaces the manual work of judging SCUs’ presence in system summaries with a natural language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role labeling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and retain SCUs that are more difficult to simulate, which provides a smooth transition and balance between automation and manual evaluation. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 existing meta-evaluation datasets and our newly collected PyrXSum (with 100/10 XSum examples/systems). It shows that Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Observations like these have led to work proposing evaluation methods that combine the advantages of human and automatic evaluation (e.g., Hashimoto et al., 2019; Zhang and Bansal, 2021).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “Thus, recent efforts have focused on aggregating model outputs and annotating quality dimensions to better assess summarization model and metric progress (Huang et al., 2020; Bhandari et al., 2020; Stiennon et al., 2020; Zhang and Bansal, 2021; Fabbri et al., 2022a; Gao and Wan, 2022).”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “Computing FACTSCORE involves (1) breaking a generation into a series of atomic facts—short statements that each contain one piece of information (Nenkova and Passonneau, 2004; Shapira et al., 2019; Zhang and Bansal, 2021; Liu et al., 2022), and (2) assigning a binary label”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “Inspired by works in summarization evaluation (Zhang and Bansal, 2021; Kamoi et al., 2023; Wang et al., 2020), we use Instruct-GPT ( text-davinci-003 ; Ouyang et al., 2022) to generate three “sub-claims”.”