Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

By Ying Xu, Xu Zhong, Antonio Jimeno-Yepes, Jey Han Lau. 2019

53 citations recorded by the source corpus.

The creation of large-scale open domain reading comprehension data sets in recent years has enabled the development of end-to-end neural comprehension models with promising results. To use these models for domains with limited training data, one of the most effective approach is to first pre-train them on large out-of-domain source data and then fine-tune them with the limited target data. The caveat of this is that after fine-tuning the comprehension models tend to perform poorly in the source domain, a phenomenon known as catastrophic forgetting. In this paper, we explore methods that reduce catastrophic forgetting during fine-tuning without assuming access to data from the source domain. We introduce new auxiliary penalty terms and observe the best performance when a combination of auxiliary penalty terms is used to regularise the fine-tuning process for adapting comprehension models. To test our methods, we develop and release 6 narrow domain data sets that can potentially be used as reading comprehension benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ting-Wei Liu
2. Minlie Huang
3. Wanxiang Che
4. Yiming Cui
5. Fei Mi
6. Liangwei Chen
7. Boi Faltings
8. Sanyuan Chen

In Computer Vision and Pattern Recognition

1. Ting-Wei Liu
2. Wanxiang Che
3. Yiming Cui
4. Minlie Huang
5. Boi Faltings
6. Sanyuan Chen
7. Minhan Kim
8. Xiangzhan Yu

Citations in the corpus

• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020, background): “, 2019; Varis and Bojar, 2019) and the reading comprehension task (Xu et al., 2019).”
• CoReD: Generalizing Fake Media Detection with Continual Representation using Distillation (2021, background): “More specifically, catastrophic forgetting (knowledge forgetting) problem [35, 39, 72, 74] from domain shifting predominantly occurs in such cases.”
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting (2020, background): “Many approaches have been proposed to overcome the forgetting problem in various domains, such as machine translation (Miceli-Barone et al., 2017; Thompson et al., 2019) and reading comprehension (Xu et al., 2019).”