Neural Domain Adaptation for Biomedical Question Answering

By Georg Wiese, Dirk Weissenborn, Mariana Neves. 2017

121 citations recorded by the source corpus.

Factoid question answering (QA) has recently benefited from the development of deep learning (DL) systems. Neural network models outperform traditional approaches in domains where large datasets exist, such as SQuAD (ca. 100,000 questions) for Wikipedia articles. However, these systems have not yet been applied to QA in more specific domains, such as biomedicine, because datasets are generally too small to train a DL system from scratch. For example, the BioASQ dataset for biomedical QA comprises less then 900 factoid (single answer) and list (multiple answers) QA instances. In this work, we adapt a neural QA system trained on a large open-domain dataset (SQuAD, source) to a biomedical dataset (BioASQ, target) by employing various transfer learning techniques. Our network architecture is based on a state-of-the-art QA system, extended with biomedical word embeddings and a novel mechanism to answer list questions. In contrast to existing biomedical QA systems, our system does not rely on domain-specific ontologies, parsers or entity taggers, which are expensive to create. Despite this fact, our systems achieve state-of-the-art results on factoid questions and competitive results on list questions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Transfer Learning in Natural Language Processing (2019, background): “…a varying portion of the pre-trained model (Long et al., 2015; Felbo et al., 2017; Howard and Ruder, 2018) with speciﬁcally designed regularization (Wiese et al., 2017; Kirkpatrick et al., 2017) or even ﬁne-tuning in sequence a model on a series of datasets using several training objectives.”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[118] Georg Wiese, Dirk Weissenborn, and Mariana Neves.”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining (2019, background): “Other deep learning based modelshave madeimprovements in biomedical text mining tasks such as relation extraction (Bhasuran and Natarajan, 2018) and question answering (Wiese et al. , 2017).”
Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models (2019, methodology): “Wiese et al. (2017) have highlighted that wdecay( w pre ) is an effective regularization technique to avoid catastrophic forgetting during ﬁnetuning.”
MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer (2020, methodology): “In fact, experimenting with methods that explicitly try to prevent catastrophic forgetting (Wiese et al., 2017) led to worse performance in our experiments.”
Improving Machine Reading Comprehension with General Reading Strategies (2018, positive): “Transfer learning techniques have been successfully applied to machine reading comprehension (Golub et al., 2017; Chung et al., 2018) and question answering (Min et al., 2017; Wiese et al., 2017).”
Cross-Lingual Transfer Learning for Question Answering (2019, background): “Wiese et al. (2017) show that it is possible to transfer knowledge from an open domain dataset for improvements on a biomedical dataset.”
Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension (2019, methodology): “For non-selective penalty, we explore L2 [8] and cosine distance.”
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (2018, background): “As a result, models which operate on these patterns (e.g. BiDAF (Seo et al., 2017)) do not generalize beyond their training domains (Wiese et al., 2017; Dhingra et al., 2018) or to novel types of reasoning (Welbl et al., 2018; Talmor and Berant, 2018).”
Natural language processing: state of the art, current trends and challenges (2017, background): “[150] introduced a deep learning approach based on domain adaptation techniques for handling biomedical question answering tasks.”
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models (2019, unknown): “We compare mix-review with theL2 regularization (weight decay) toward the pre-trained parameters θpre (Wiese et al., 2017).”
BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR (2020, methodology): “And like (Lee et al., 2019) and (Wiese et al., 2017), we ﬁne-tuned BioASQ on weights that were initially ﬁne-tuned on SQuAD v1.”
Multi-Stage Pretraining for Low-Resource Domain Adaptation (2020, background): “In cases where the target task has limited labeled data, prior work has also employed transfer learning by pre-training on a source dataset with abundant labeled data before ﬁne-tuning on the target task dataset (Min et al., 2017; Chung et al., 2018; Wiese et al., 2017).”