An Information-Theoretic Approach to Automatic Evaluation of Summaries

By Chin-Yew Lin, Guihong Cao, Jianfeng Gao, Jian-Yun Nie. 2006

106 citations recorded by the source corpus.

Until recently there are no common, convenient, and repeatable evaluation methods that could be easily applied to support fast turn-around development of automatic text summarization systems. In this paper, we introduce an information-theoretic approach to automatic evaluation of summaries based on the Jensen-Shannon divergence of distributions between an automatic summary and a set of reference summaries. Several variants of the approach are also considered and compared. The results indicate that JS divergence-based evaluation method achieves comparable performance with the common automatic evaluation method ROUGE in single documents summarization task; while achieves better performance than ROUGE in multiple document summarization task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “For the AB format, we use ROUGE-L metric (Lin et al., 2006; Min et al., 2019; Nishida et al., 2019).”
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge (2021, methodology): “For ROUGE (Lin et al., 2006), we use the F1 score for the ROUGE-L variant which considers the longest common subsequence, thus penalizing words out of order.7 For the simple F1 word-overlap measure, we adopt the conventions from the SQuAD dataset (Rajpurkar et al., 2016) in terms of ignoring…”
Exploring Content Models for Multi-Document Summarization (2009, background): “Lin et al. (2006) propose a related criterion for robust summarization evaluation, but to our knowledge this criteria has been unexplored in summarization systems.”
Learning to Score System Summaries for Better Content Selection Evaluation. (2017, background): “only on the candidate summary and the source documents (Lin et al., 2006; Louis and Nenkova, 2013).”
A Survey on Automatic Text Summarization (2007, methodology): “(26)Although this is not mentioned in (Lin et al., 2006), the Jensen-Shannon divergence also satisfies the axioms to be a squared metric, as shown by Endres and Schindelin (2003).”
A Survey of Text Summarization Techniques (2012, result): “This intuition has been studied extensively in work on automatic evaluation of content selection in summarization, where another indicator of divergence—Jensen Shannon divergence—has proven superior to KL [45, 52].”
Studying Summarization Evaluation Metrics in the Appropriate Scoring Range (2019, methodology): “JS divergence ( JS-2 ): uses Jensen-Shannon divergence between bigram distributions of references and candidate summaries (Lin et al., 2006).”
G ENIE A Leaderboard for Human-in-the-Loop Evaluation of Text Generation (2021, methodology): “In addition, G ENIE presents various automatic metrics (e.g., BLEU, Papineni et al., 2002; ROUGE, Lin, 2004) to examine how well they correlate with the human assessment scores.”
Re-evaluating Evaluation in Text Summarization (2020, background): “JS divergence (JS-2) measures Jensen-Shannon divergence between the two text’s bigram distributions7 (Lin et al., 2006).”
Finding a Balanced Degree of Automation for Summary Evaluation (2021, background): “Some other works compute the similarity over n-gram graphs (Giannakopoulos and Karkaletsis, 2011; Giannakopoulos et al., 2008) or distributions (Lin et al., 2006).”
Objective Function Learning to Match Human Judgements for Optimization-Based Summarization (2018, background): “We also report JS2, the Jensen-Shannon divergence between bigrams in the reference summaries and the candidate system summary (Lin et al., 2006).”
CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information (2022, methodology): “Test-Acc can compensate for shortcomings of Bleu [28] value, Rouge-L [38], StrAcc and ACC+ [10] in evaluating cases 2 and 3 described in 3.2.”
Multilingual Summarization Evaluation without Human Models (2010, methodology): “Lin et al. (2006) proposed a method of evaluation based on the use of “distances” or divergences between two probability distributions (the distribution of units in the automatic summary and the distribution of units in the model summary).”
Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics (2020, background): “JS divergence (JS-2) measures Jensen-Shannon divergence between the two text’s bigram distributions5 (Lin et al., 2006).”