Fill in the BLANC: Human-free quality estimation of document summaries

By Oleg V. Vasilyev, Vedant Dharnidharka, John Bohannon. 2020

147 citations recorded by the source corpus.

We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document’s text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “[158] propose BLANC that defined by how well a summary helps an independent, pre-trained language model while it performs its language understanding task on the input document.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “While BLANC-Help, FEQA metric and BERTScore values decrease with factual inconsistency on controlled data, the metrics may some-times be positively correlated with factual inconsistency on generated data.”
Best Practices for Crowd-based Evaluation of German Summarization: Comparing Crowd, Expert and Automatic Evaluation (2020, methodology): “Much more complex annotation aggregation methods such as probabilistic models of annotation, accounting item level effects, or clustering methods have been introduced in the recent years (Passonneau and Carpenter, 2014; Whitehill et al., 2009; Luther et al., 2015).”
TuringAdvice: A Generative and Dynamic Evaluation of Language Use (2021, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “Miscellaneous Besides the above metrics, we also use competitive metrics including UniEval (Zhong et al., 2022), CTC (Deng et al., 2021), BARTScore (Yuan et al., 2021) and BLANC (Vasilyev et al., 2020) as baselines.”
Evaluating Machines by their Real-World Language Use (2020, methodology): “Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020).”
News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “, 2020), which evaluates generated summaries against automatically identified salient sentences in the input, and BLANC (Vasilyev et al., 2020), which evaluates summaries on language un-”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, unknown): “Compared to the results of Fabbri et al. (2021b), metrics based on n-gram overlap such as ROUGE and CHRF perform worse on dialogue summarization, while some metrics that use source documents such as BLANC perform better.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, unknown): “…0.02 0.11 0.09 0.34 0.02 0.56 0.26 0.09 0.32 0.48 0.31 0.29 0.53 0.03 0.18 0.11 0.18 0.02 0.58 0.27 0.09 (f) Image Captioning proposed metrics such as DEB (Sai et al., 2020a), BLANC (Vasilyev et al., 2020) and MaUde (Sinha et al., 2020) do not correlate well with human judgements on other criteria.”
Does Summary Evaluation Survive Translation to Other Languages? (2021, methodology): “131 We calculate the following truly automated (not 132 needing human written reference summaries) mea- 133 sures: Jensen-Shannon (Louis and Nenkova, 2009), 134 ESTIME (Vasilyev and Bohannon, 2021)4 and 135 BLANC (Vasilyev et al., 2020)5.”
SummEval: Re-evaluating Summarization Evaluation (2020, background): “BLANC (Vasilyev et al., 2020) is a referenceless metric that measures the performance gains of a pre-trained language model given access to a document summary while carrying out language understanding tasks on the source document’s text.”
Benchmarking Large Language Models for News Summarization (2023, methodology): “For CNN/DM, we observe that the reference-based automatic metrics have a moderate correlation with some aspects of human judgments, e.g., Rouge-L has a 0.72 Kendall’s tau correlation co-efﬁcient with relevance in Table 3.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “BLANC (Vasilyev et al., 2020) measures the performance gains of a pre-trained language model on language understanding tasks on the input document when given access to a document summary.”