Better Metrics to Automatically Predict the Quality of a Text Summary
By Peter A. Rankel, John M. Conroy, Judith D. Schlesinger. 2012
20 citations recorded by the source corpus.
In this paper we demonstrate a family of metrics for estimating the quality of a text summary relative to one or more human-generated summaries. The improved metrics are based on features automatically computed from the summaries to measure content and linguistic quality. The features are combined using one of three methods—robust regression, non-negative least squares, or canonical correlation, an eigenvalue method. The new metrics significantly outperform the previous standard for automatic text summarization evaluation, ROUGE.
Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Elizabeth Clark
- Sebastian Gehrmann
- Thibault Sellam
- Ani Nenkova
- Hoa Trang Dang
- John M. Conroy
- Peter A. Rankel
- Yejin Choi
In Natural Language Processing
- Elizabeth Clark
- Thibault Sellam
- Sebastian Gehrmann
- Hoa Trang Dang
- John M. Conroy
- Peter A. Rankel
- Ani Nenkova
- Maxime Peyrard
In Evaluation & Benchmarks
- Sebastian Gehrmann
- Elizabeth Clark
- Thibault Sellam
- Hoa Trang Dang
- John M. Conroy
- Peter A. Rankel
- Ani Nenkova
- Maxime Peyrard
Citations in the corpus
- Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “The first metric of this kind was CLASSY, a logistic regression model for summarization evaluation (Rankel et al., 2012).”
- A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art (2013, background): “This approach was extended by Rankel et al. (2012) by including measures of linguistic quality.”