SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization
295 citations recorded by the source corpus.
In this paper, we present a conceptually simple while empirically powerful framework for abstractive summarization, SimCLS, which can bridge the gap between the learning objective and evaluation metrics resulting from the currently dominated sequence-to-sequence learning framework by formulating text generation as a reference-free evaluation problem (i.e., quality estimation) assisted by contrastive learning. Experimental results show that, with minor modification over existing top-scoring systems, SimCLS can improve the performance of existing top-performing models by a large margin. Particularly, 2.51 absolute improvement against BART and 2.50 over PEGASUS w.r.t ROUGE-1 on the CNN/DailyMail dataset, driving the state-of-the-art performance to a new level. We have open-sourced our codes and results: https://github.com/yixinL7/SimCLS. Results of our proposed models have been deployed into ExplainaBoard platform, which allows researchers to understand our systems in a more fine-grained way.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Representation & Self-Supervised Learning
In Summarization
Citations in the corpus
- BARTScore: Evaluating Generated Text as Text Generation (2021, background): “BART [27] as our backbone due to its superior performance in text generation [8, 34, 60].”
- ExplainaBoard: An Explainable Leaderboard for NLP (2021, methodology): “System outputs on text summarization are expanded based on the previous work’s collection (Bhandari et al., 2020) as well as recently state-of-the-art systems (Liu and Liu, 2021) while outputs from machine translation are collected from the WMT20.”
- BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “Some recent works (Liu and Liu, 2021; Lee et al., 2021a) have also explored discriminative reranking of candidates from neural natural language generation models, which adopt large pre-trained language models (e.g. BERT (Devlin et al., 2019)) as the reranker.”
- LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023, methodology): “SimCLS (Liu and Liu, 2021) encodes the input x and each generated candidate yi ∈ Y using the same encoder H , resulting in H(x) and H(yi).”
- CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization (2021, methodology): “Using a similar idea, Liu and Liu (2021) enlarge the distances among summaries of different quality as measured by ROUGE scores.”
- FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization (2022, methodology): “To tackle this problem, many approaches propose post-processing models (Chen et al., 2021; Dong et al., 2020; Liu and Liu, 2021), but such methods are often constrained by external resources to train additional correction or selection models.”
- Language Models Can See: Plugging Visual Controls in Text Generation (2022, background): “Beyond representation learning, contrastive learning has also been applied to other NLP applications such as name entity recognition (NER) [13] and document summarisation [45], knowledge probing for pre-trained language models [50], and open-ended text generation [72].”
- Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization (2021, background): “With the rapid development of neural networks in text summarization (Liu and Lapata, 2019; Liu, 2019; Zhong et al., 2019; Zhang et al., 2019; Lewis et al., 2019; Zhong et al., 2020; Liu and Liu, 2021), especially the use of contextualized pre-trained models (Devlin et al.”