AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization

By Tiezheng Yu, Zihan Liu, Pascale Fung. 2021

87 citations recorded by the source corpus.

State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstractive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model’s catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Andrea Madotto
3. Wenliang Dai
4. Zihan Liu
5. Nayeon Lee
6. Yan Xu
7. Etsuko Ishii
8. Ziwei Ji

In Natural Language Processing

1. Pascale Fung
2. Nayeon Lee
3. Andrea Madotto
4. Zihan Liu
5. Etsuko Ishii
6. Yejin Bang
7. Yan Xu
8. Rita Frieske

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “Abstractive summarization aims to extract essential information from source documents and to generate short, concise, and readable summaries [158].”
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity (2023, background): “Interactivity on Summarization Summarization models aim to extract essential information from documents and to generate short, concise, and readable text (Yu et al., 2021b; Su et al., 2021).”
• Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021, background): “, 2014) and attention mechanisms, deep neural networks have achieved remarkable results on summarization tasks (Paulus et al., 2017; Zhang et al., 2020b; Yu et al., 2021).”