How to Compare Summarizers without Target Length? Pitfalls, Solutions and Re-Examination of the Neural Summarization Literature

By Simeng Sun, Ori Shapira, Ido Dagan, Ani Nenkova. 2019

53 citations recorded by the source corpus.

We show that plain ROUGE F1 scores are not ideal for comparing current neural systems which on average produce different lengths. This is due to a non-linear pattern between ROUGE F1 and summary length. To alleviate the effect of length during evaluation, we have proposed a new method which normalizes the ROUGE F1 scores of a system by that of a random system with same average output length. A pilot human evaluation has shown that humans prefer short summaries in terms of the verbosity of a summary but overall consider longer summaries to be of higher quality. While human evaluations are more expensive in time and resources, it is clear that normalization, such as the one we proposed for automatic evaluation, will make human evaluations more meaningful.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “For each task, we additionally report the vocabulary size over the output (|V|) and the mean output length of a system (Sun et al., 2019).”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, background): “Sun et al. (2019) recently show that, for summaries whose lengths are in the range of 50 to 110 tokens, longer summaries receive higher ROUGE-F1 scores.”
Hurdles to Progress in Long-form Question Answering (2021, positive): “(Sun et al., 2019) has also shown that ROUGE scores vary heavily by length.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “related work (Sun et al., 2019; Song et al., 2021; Gehrmann et al., 2022b; Goyal et al., 2022a) has shown that summary length can have a non-trivial impact on the model performance and evaluation.”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, background): “Despite our generally superior results on the ROUGE metric, past work (Schluter, 2017; Hoang et al., 2019; Sun et al., 2019) has explored the limitations of ROUGE for evaluating summarization tasks, including the issue that ROUGE measures n -gram overlap without distinguishing between salient n -gra”
Generating Abstractive Summaries with Finetuned Language Models (2019, background): “Despite the added inference-time loss functions, the LSTM models consistently perform worse than 1 We note that an increased length of generated summaries has been found to increase ROUGE scores which make comparison to other systems with different length outputs challenging (Sun et al., 2019).”
Efficient Adaptation of Pretrained Transformers for Abstractive Summarization (2019, result): “First, echoing the results of Sun et al. [24], the most common summarization evaluation metric, ROUGE [10], is highly sensitive to summary length, providing an advantage to methods that produce longer summaries, either through learning or with minimum summary length constraints.”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, background): “…judgments (Papineni et al., 2002; Doddington, 2002; Coughlin, 2003) for concurrent models; though, the correspondence breaks down as they become targets for optimization (Callison-Burch et al., 2006; Sun et al., 2019) or as models become increasingly powerful (Ma et al., 2019; Edunov et al., 2020).”