UvA-DARE (Digital Academic Repository) A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs

By Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi

52 citations recorded by the source corpus.

We introduce a manually-created, multi-reference dataset for abstractive sentence and short paragraph compression. First, we examine the impact of single- and multi-sentence level editing operations on human compression quality as found in this corpus. We observe that substitution and rephrasing operations are more meaning preserving than other operations, and that compressing in context improves quality. Second, we systematically explore the correlations between automatic evaluation metrics and human judgments of meaning preservation and grammaticality in the compression task, and analyze the impact of the linguistic units used and precision versus recall measures on the quality of the metrics. Multi-reference evaluation metrics are shown to offer significant advantage over single reference-based metrics.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. M. Zhou
3. Mohit Bansal
4. Sascha Rothe
5. Jianfeng Gao
6. Aliaksei Severyn
7. Nan Yang
8. Mirella Lapata

Citations in the corpus

• Encode, Tag, Realize: High-Precision Text Editing (2019, methodology): “Out of available summarization datasets (Dernoncourt et al., 2018), we find the one by Toutanova et al. (2016) particularly interesting because (1) it specifically targets abstractive summarization systems, (2) the lengths of texts in this dataset (short paragraphs) seem well-suited for text…”
• FELIX: Flexible Text Editing Through Tagging and Insertion (2020, methodology): “We use the dataset from (Toutanova et al., 2016), which contains 6,168 short input texts (one or two sentences) and one or more human-written”
• A New Annotated Portuguese/Spanish Corpus for the Multi-Sentence Compression Task (2018, methodology): “Cohn and Lapata (2008) and Toutanova et al. (2016) describe two manually created ab-stractive compression corpora that are publicly available.”
• A Continuously Growing Dataset of Sentential Paraphrases (2017, background): “…and Elhadad, 2003; Barzilay and Lee, 2003) or other texts (Cohn et al., 2008), multiple definitions of the same concept (Hashimoto et al., 2011), descriptions of the same video clip from multiple workers (Chen and Dolan, 2011) or rephrased sentences (Burrows et al., 2013; Toutanova et al., 2016).”
• Selective Encoding for Abstractive Sentence Summarization (2017, methodology): “DUC 2004 Test Set We employ DUC 2004 data for tasks 1 & 2 (Over et al., 2007) Table 2 summarizes the statistic information of the three datasets we used.”
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression (2024, background): “For Q1, most text compression datasets are ab-stractive (Toutanova et al., 2016; Koupaee and Wang, 2018; Kim et al., 2019), meaning that they treat prompt compression as a generative task where the original prompts are rephrased into condensed ones.”
• Split and Rephrase (2017, background): “As noted by Pitler (2010) and Toutanova et al. (2016) however, the ability to paraphrase is key for the development of abstractive summarisation systems since summaries written by humans often rephrase the original content using paraphrases or synonyms or alternative syntactic constructions.”
• Recent advances in document summarization (2017, unknown): “A recent study [179] introduces a manually-created”