Multi-Stage Pretraining for Low-Resource Domain Adaptation

By Rong Zhang, Revanth Reddy Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avirup Sil, Tomás Ward. 2020

48 citations recorded by the source corpus.

Transfer learning techniques are particularly useful in NLP tasks where a sizable amount of high-quality annotated data is difficult to obtain. Current approaches directly adapt a pre-trained language model (LM) on in-domain text before fine-tuning to downstream tasks. We show that extending the vocabulary of the LM with domain-specific terms leads to further gains. To a bigger effect, we utilize structure in the unlabeled data to create auxiliary synthetic tasks, which helps the LM transfer to downstream tasks. We apply these approaches incrementally on a pre-trained Roberta-large LM and show considerable performance gain on three tasks in the IT domain: Extractive Reading Comprehension, Document Ranking and Duplicate Question Detection.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Robust Neural Retrieval Models with Synthetic Pre-Training (2021, methodology): “Inspired by the success of synthetic training examples in MRC (Shakeri et al., 2020; Zhang et al., 2020; Sultan et al., 2020), we further investigate if a similar approach can be useful for IR, including zero-shot application settings.”
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting (2022, unknown): “…(restricted to the embedding layer training only in some cases) (Neubig and Hu, 2018; Artetxe et al., 2020; Chau et al., 2020; Muller et al., 2021; Zhang et al., 2020; Wang et al., 2020); (2) training of language-specific adapters (Pfeiffer et al., 2020, 2021a,b; Philip et al., 2020; Üstün et…”