CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

By Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, Dragomir R. Radev. 2021

72 citations recorded by the source corpus.

Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained neural language models, substantial amounts of hallucinated content are found during the human evaluation. In this work, we first devised a typology of factual errors to better understand the types of hallucinations generated by current models and conducted human evaluation on popular dialog summarization dataset. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called CONFIT. To tackle top factual errors from our annotation, we introduce additional contrastive loss with carefully designed hard negative samples and self-supervised dialogue-specific loss to capture the key information between speakers. We show that our model significantly reduces all kinds of factual errors on both SAMSum dialogue summarization and AMI meeting summarization. On both datasets, we achieve significant improvements over state-of-the-art baselines using both automatic metrics, ROUGE and BARTScore, and human evaluation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Andrea Madotto
3. Wenliang Dai
4. Nayeon Lee
5. Yan Xu
6. Etsuko Ishii
7. Ziwei Ji
8. Dan Su

In Natural Language Processing

1. Pascale Fung
2. Nayeon Lee
3. Andrea Madotto
4. Etsuko Ishii
5. Yejin Bang
6. Yan Xu
7. Rita Frieske
8. Dan Su

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “[179] propose another contrastive fine-tuning strategy, named CONFIT, that can improve the factual consistency and overall quality of summaries.”
• DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, methodology): “Many studies, such as Chen and Yang (2020) and Tang et al. (2021), have pointed out that the current dialogue summarization models still have many shortcomings, such as wrong references, incorrect reasoning and improper gender pronouns, and ROUGE may not reflect these problems effectively.”