SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization

By Yao Zhao, Mohammad Saleh, Peter J. Liu. 2020

28 citations recorded by the source corpus.

Most prior work in the sequence-to-sequence paradigm focused on datasets with input sequence lengths in the hundreds of tokens due to the computational constraints of common RNN and Transformer architectures. In this paper, we study long-form abstractive text summarization, a sequence-to-sequence setting with input sequence lengths up to 100,000 tokens and output sequence lengths up to 768 tokens. We propose SEAL, a Transformer-based model, featuring a new encoder-decoder attention that dynamically extracts/selects input snippets to sparsely attend to for each output segment. Using only the original documents and summaries, we derive proxy labels that provide weak supervision for extractive layers simultaneously with regular supervision from abstractive summaries. The SEAL model achieves state-of-the-art results on existing long-form summarization tasks, and outperforms strong baseline models on a new dataset/task we introduce, Search2Wiki, with much longer input text. Since content selection is explicit in the SEAL model, a desirable side effect is that the selection can be inspected for enhanced interpretability.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Greg Durrett
2. Mohit Bansal
3. Percy Liang
4. Junyi Jessy Li
5. Jan Leike
6. Jeff Wu
7. Heng Ji
8. Tanya Goyal

In Molecular Biology

1. Heng Ji
2. Greg Durrett
3. Lu Wang
4. Junyi Jessy Li
5. Tanya Goyal

In Natural Language Processing

1. Greg Durrett
2. Junyi Jessy Li
3. Tanya Goyal
4. Lu Wang
5. Shuyang Cao
6. Heng Ji
7. Paul Christiano
8. Nikolaus Nova Parulian

Citations in the corpus

• Recursively Summarizing Books with Human Feedback (2021, methodology): “…final summaries, either by having a hierarchical encoder (Cohan et al., 2018; Zhang et al., 2019c; Liu and Lapata, 2019a), or by first running an extractive summarization model followed by an abstractive model (Subramanian et al., 2019; Liu et al., 2018; Zhao et al., 2020; Gharebagh et al., 2020).”
• Efficient Attentions for Long Document Summarization (2021, methodology): “…2019), a large pre-trained summarization model with truncated inputs; TLM (Pilault et al., 2020), DANCER (Gidiotis and Tsoumakas, 2020), and SEAL (Zhao et al., 2020), all of them using hybrid extract-then-abstract methods; and B IG B IRD (Za-heer et al., 2020), which combines sliding window,…”
• News Summarization and Evaluation in the Era of GPT-3 (2022, methodology): “16 Next, we follow the commonly used segment-then-summarize procedure from prior work (Zhao et al., 2020; Zhang et al., 2022).”