TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

By Siddhant Garg, Thuy Vu, Alessandro Moschitti. 2019

233 citations recorded by the source corpus.

We propose TandA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving the impressive MAP scores of 92% and 94.3%, respectively, which largely outperform the the highest scores of 83.4% and 87.5% of previous work. We empirically show that TandA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TandA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TandA in an industrial setting, using domain specific datasets subject to different types of noise.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “des the candidates to be reranked (i.e., ﬁrst-stage retrieval is ﬁxed and part of the test collection itself). Results copied from the authors’ paper are shown in Table 14. The baseline is TANDABASE [Garg et al., 2020], which is monoBERT with a multi-stage ﬁne-tuning procedure that uses multiple dat”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “• Two-stage ﬁne-tuning introduces an intermediate supervised training stage between pre-training and ﬁne-tuning (Phang et al., 2019; Garg et al., 2020).”
ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, methodology): “For example, TANDA (Garg et al., 2019) adopts an intermediate training on modified Natural Questions dataset (Kwiatkowski et al.”
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “Garg and Moschitti [31] (2021) Text ranking Question answering WikiQA [163], ASNQ [32] SQuAD 1.”
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (2020, methodology): “Fine-tuning of large pre-trained trans-formers such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) currently achieves state-of-the-art performances on many related benchmarks (Garg et al., 2020; Mass et al., 2019; Ro-chette et al., 2019; Nogueira and Cho, 2019).”
Answer Generation for Retrieval-based Question Answering Systems (2021, methodology): “…contain 126 and 243 unique questions and we reannotated all of the resulting 569 QA pairs.2 Answer Sentence Natural Questions (ASNQ) introduced by Garg et al. (2020) was derived from the NQ dataset (Kwiatkowski et al., 2019) and consists of the questions which have a short answer span within a…”
Reranking for Efficient Transformer-based Answer Selection (2020, background): “Questions (ASNQ): Famous benchmarks for AS2 such as TREC-QA [20] and WikiQA [24] do not come with an large enough number of annotated candidates to simulate a real application scenario.”
AVA: an Automatic eValuation Approach to Question Answering Systems (2020, methodology): “Thus, we retrieved high-probably relevant candidates as follows: we (i) retrieved top 500 relevant documents; (ii) automatically extracted the top 100 sentences ranked by a BERT model over all sentences of the documents; and (iii) had all the top 100 sentences manually annotated as correct or incorr”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, methodology): “When applicable 8 , we ﬁne-tune our models using the two-step “transfer and adapt” (T AND A) technique introduced by Garg et al. (2020).”
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023, background): “For instance, tasks in the Biomedical and the Clinical domain (Luo et al., 2022; Lee et al., 2020; Alsentzer et al., 2019; Beltagy et al., 2019; Gu et al., 2020; Peng et al., 2019), NER across more datasets (Tjong Kim Sang and De Meulder, 2003; Malmasi et al., 2022; Fu et al., 2022; Laskar et al., 2”
Deep Learning--based Text Classification (2020, background): “BERT and its variants have been fine-tuned for various NLP tasks, including QA [153], TC [154], and NLI [23, 155].”
Open-Retrieval Conversational Question Answering (2020, background): “The choices of Kr t are [16, 50, 100, 500, 1000].”
A large language model for electronic health records (2022, methodology): “16 In the general domain, the transformer-based NLP models have achieved state-of-the-art performance for name entity recognition 17–19 , relation extraction 20–24 , sentence similarity 25–27 , natural language inference 27–30 , and question answering 27,28,31,32 .”