SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization

By Philippe Laban, Tobias Schnabel, Paul N. Bennett, Marti A. Hearst. 2021

574 citations recorded by the source corpus.

In the summarization domain, a key requirement for summaries is to be factually consistent with the input document. Previous work has found that natural language inference (NLI) models do not perform competitively when applied to inconsistency detection. In this work, we revisit the use of NLI for inconsistency detection, finding that past work suffered from a mismatch in input granularity between NLI datasets (sentence-level), and inconsistency detection (document level). We provide a highly effective and light-weight method called SummaCConv that enables NLI models to be successfully used for this task by segmenting documents into sentence units and aggregating scores between pairs of sentences. We furthermore introduce a new benchmark called SummaC (Summary Consistency) which consists of six large inconsistency detection datasets. On this dataset, SummaCConv obtains state-of-the-art results with a balanced accuracy of 74.4%, a 5% improvement compared with prior work.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “…et al., 2015), SPICE (Anderson et al., 2016) (for captioning/commonsense generation), PARENT (Dhingra et al., 2019) (for data-to-text) and SummaCZS (Laban et al., 2022) (for factuality of summarization); 4) diversity/fluency/naturalness metrics such as perplexity, Mean Segmented Type Token Ratio…”
Survey of Hallucination in Natural Language Generation (2022, methodology): “These metrics are based on the idea that only the source knowledge reference should entail the entirety of the information in faithful and hallucination-free generation [28, 31, 32, 54, 57, 67, 68, 92].”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, methodology): “We follow Laban et al. (2021) and perform signiﬁcance testing through bootstrap resampling (Efron, 1982), comparing the best method to the second-best method on each dataset.”
Large Language Model Alignment: A Survey (2023, background): “While this could be done on a single task (Laban et al., 2022; Fabbri et al., 2021), consistency evaluation on multiple tasks is more convincing.”
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function (2023, methodology): “We speculate that it is because 1) premises and hypothesises in NLI datasets are generally shorter, which differs from most factual consistency benchmarks and datasets, 2) other NLP tasks have larger-scale and higher quality datasets.”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, methodology): “Knowledge Conflicts We evaluate performance on two knowledge conflict datasets: MemoTrap (Liu and Liu, 2023) and NQ-Swap (Longpre et al., 2021).”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “This includes model-based evaluation in summarization that considers the consistency between a summary and a source document using QA or NLI (Kryscin-ski et al., 2020; Wang et al., 2020; Fabbri et al., 2022; Deutsch et al., 2021; Laban et al., 2022).”
Towards a Unified Multi-Dimensional Evaluator for Text Generation (2022, methodology): “For the score calculation, we follow previous work to compute sentence-level average scores for fluency and consistency (Laban et al., 2021) in summarization, and sentence-level cumulative scores for engagingness (Deng et al.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “…have proposed methods such as fine-tuning on adversarial datasets (Barrantes et al., 2020), decomposing the entailment decisions at the dependency arc level (Goyal and Durrett, 2020), and segmenting documents into sentence units then aggregating scores between sentence pairs (Laban et al., 2022).”
Benchmarking Large Language Models for News Summarization (2023, methodology): “The list of metrics we evaluate are: Rouge-L (Lin, 2004), METEOR (Baner-jee and Lavie, 2005), BertScore (Zhang* et al., 2020), BLEURT (Sellam et al., 2020), and BARTScore (Yuan et al., 2021). on CNN/DM and XSUM so we discuss them separately in the following paragraphs.”
Detecting hallucinations in large language models using semantic entropy (2024, unknown): “Textual entailment has previously been shown to correlate with faithfulness 10 in the context of factual consistency 29 as well as being used to measure factuality in abstractive summarization 30 , especially when applied at the right granularity 31 .”
"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters (2023, methodology): “Our study utilizes NLI-based approach (Kryscinski et al., 2020; Maynez et al., 2020; Laban et al., 2022), which uses the original input as context and determine the entailment of model-generated”
News Summarization and Evaluation in the Era of GPT-3 (2022, unknown): “, 2020), DAE (Goyal and Durrett, 2020, 2021) and SummaC (Laban et al., 2022).”
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, methodology): “We refer the reader to Laban et al. (2021) for further details regarding the benchmark creation.”
SMART: Sentences as Basic Units for Text Evaluation (2022, background): “…metrics can be divided into three approaches: nat-ural language inference (NLI) based approaches (Falke et al., 2019; Maynez et al., 2020; Laban et al., 2022), where a summary is considered factual if all its facts are entailed by the source document, model-based approaches (Kryscinski…”