SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization

By Bogdan Gliwa, Iwona Mochol, Maciej Biesek, Aleksander Wawer. 2019

841 citations recorded by the source corpus.

This paper introduces the SAMSum Corpus, a new dataset with abstractive dialogue summaries. We investigate the challenges it poses for automated summarization by testing several models and comparing their results with those obtained on a corpus of news articles. We show that model-generated summaries of dialogues achieve higher ROUGE scores than the model-generated summaries of news – in contrast with human evaluators’ judgement. This suggests that a challenging task of abstractive dialogue summarization requires dedicated models and non-standard quality measures. To our knowledge, our study is the first attempt to introduce a high-quality chat-dialogues corpus, manually annotated with abstractive summarizations, which can be used by the research community for further studies.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, neutral): “Gao and Wan (2022) applied the same annotation scheme to the SAMSum dialog summarization dataset (Gliwa et al., 2019) and confirmed that these findings are consistent across datasets.”
Finetuned Language Models Are Zero-Shot Learners (2021, neutral): “Samsum (Gliwa et al., 2019) 7.”
UL2: Unifying Language Learning Paradigms (2022, methodology): “We use CNN/Dailymail (Hermann et al., 2015), XSUM (Narayan et al., 2018), MultiNews (Fabbri et al., 2019), SAMSum (Gliwa et al., 2019), WebNLG (Castro Ferreira et al., 2020) (English), E2E (Dušek et al., 2019) and CommonGen (Lin et al., 2020) to evaluate our models.”
Survey of Hallucination in Natural Language Generation (2022, background): “[59], it would be beneficial to apply other works which are aiming to grasp the longer context but do not focus on dialogue generation [8, 223, 233].”
LoRA: Low-Rank Adaptation of Large Language Models (2021, background): “Specifically, we focus on WikiSQL [40] (natural language to SQL queries), MultiNLI [37] (natural language inference) 4, and SAMSum [12] (conversation summarization) for GPT-3.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…et al. 2019 ropes qa/machine reading comprehension Lin et al. 2019 rotten_tomatoes cls/sentiment analysis Pang and Lee 2005 samsum cg/summarization Gliwa et al. 2019 scicite cls/other Cohan et al. 2019 sciq qa/multiple-choice qa Welbl et al. 2017 scitail cls/nli Khot et al. 2018 search_qa…”
Beyond Goldfish Memory: Long-Term Open-Domain Conversation (2021, background): “, 2019), while there is some work on dialogue as well (Goo and Chen, 2018; Gliwa et al., 2019; Pan et al., 2018).”
Language Models are General-Purpose Interfaces (2022, background): “• Summarization is to generate a summary of a given passage: AESLC (Zhang and Tetreault, 2019), SamSum (Gliwa et al., 2019), XSum (Narayan et al., 2018).”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, methodology): “With this goal in mind, we design a dataset in the following way: First, we take a LightWild episode and use an abstractive summarization model, trained on CNN/Daily Mail (Nallapati et al., 2016) and the SAMSum Corpus (Gliwa et al., 2019), to generate a summary.”
GO FIGURE: A Meta Evaluation of Factuality in Summarization (2020, methodology): “…summarization dataset (Narayan et al., 2018), multi-sentence summaries from the CNN/DailyMail dataset (Nallapati et al., 2016), and the recently released SAMSUM corpus (Gliwa et al., 2019) consisting of English language conversations written by linguists and aligned multi-sentence summaries.”
Planning with Learned Entity Prompts for Abstractive Summarization (2021, background): “, 2018), SAMSum dialogue summarization (Gliwa et al., 2019), and BillSum summarizing US Congressional bills (Kornilova and Eidelman, 2019).”
Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques (2020, background): “…is much harder than traditional summarization benchmarks, in part, because SOAP notes are longer (320 words on average) than summaries in popular datasets like CNN/Dailymail (Nallapati et al., 2016), Newsroom (Grusky et al., 2018), and Sam-Sum (Gliwa et al., 2019) (55, 27, and 24 words on average).”
Time Travel in LLMs: Tracing Data Contamination in Large Language Models (2023, methodology): “The datasets in question involve IMDB (Maas et al. 2011), AG News (Zhang et al. 2015), Yelp Full Reviews (Zhang et al. 2015), SAMSum (Gliwa et al. 2019), XSum (Narayan et al. 2018), WNLI (Wang et al. 2018), and RTE (Wang et al. 2019).”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, methodology): “, 2018), and SamSum (Gliwa et al., 2019), and annotations”
DialSummEval: Revisiting Summarization Evaluation for Dialogues (2022, methodology): “With the proposal of dialogue summary datasets such as SAMSum (Gliwa et al., 2019), DialogSum (Chen et al.”
BookSum: A Collection of Datasets for Long-form Narrative Summarization (2021, background): “Gliwa et al. (2019) introduced a corpus of conversations between hired annotators designed to mimic interactions on a messaging application with human written summaries.”