CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization

By Shuyang Cao, Lu Wang. 2021

212 citations recorded by the source corpus.

We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Survey of Hallucination in Natural Language Generation (2022, background): “Cao and Wang [18] introduce a contrastive learning method to train summarization models.”
BRIO: Bringing Order to Abstractive Summarization (2022, background): “, 2021), text summarization (Cao and Wang, 2021; Xu et al., 2021; Sun and Li, 2021), and other tasks (Uehara et al.”
FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “Cao and Wang (2021) extend this idea to improve factu-ality with various approaches to generate halluci-nated summaries as negative examples, showing consistent improvement over existing methods.”
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding (2023, background): “, 2021), optimizing summarization models towards a factual consistency metrics (Nan et al., 2021; Cao and Wang, 2021),”
Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling (2022, background): “Model designs and factuality specific objectives help optimize for factuality during training (Gabriel et al., 2019; Cao and Wang, 2021; Dong et al., 2022; Rajagopal et al., 2022).”