MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale

By Andreas Ruckl'e, Jonas Pfeiffer, Iryna Gurevych. 2020

39 citations recorded by the source corpus.

We study the zero-shot transfer capabilities of text matching models on a massive scale, by self-supervised training on 140 source domains from community question answering forums in English. We investigate the model performances on nine benchmarks of answer selection and question similarity tasks, and show that all 140 models transfer surprisingly well, where the large majority of models substantially outperforms common IR baselines. We also demonstrate that considering a broad selection of source domains is crucial for obtaining the best zero-shot transfer performances, which contrasts the standard procedure that merely relies on the largest and most similar domains. In addition, we extensively study how to best combine multiple source domains. We propose to incorporate self-supervised with supervised multi-task learning on all available source domains. Our best zero-shot transfer model considerably outperforms in-domain BERT and the previous state of the art on six benchmarks. Fine-tuning of our model with in-domain data results in additional large gains and achieves the new state of the art on all nine benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, neutral): “Adapterbased approaches have been proposed as a remedy (Rebuffi et al., 2017, 2018; Houlsby et al., 2019; Stickland and Murray, 2019; Bapna and Firat, 2019; Pfeiffer et al., 2020a,b; Lauscher et al., 2020a; Rücklé et al., 2020a,b, inter alia).”
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer (2020, background): “We will also explore the usefulness of orthoadapters in other tasks as well as for domain adaption (R¨uckl´e et al., 2020).”
Cross-Lingual Training of Neural Models for Document Ranking (2020, unknown): “R¨uckl´e et al. (2020).”
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021, result): “This indicates re-ranking methods involving a cross-encoder generalize well to unseen datasets and domains, which is in line with the findings of other recent work (Akkalyoncu Yilmaz et al., 2019; Rücklé et al., 2020; Thakur et al., 2020).”
Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision (2021, unknown): “, 2007) for different NLP tasks by designing proper pretext tasks (Wang et al., 2019; Wu et al., 2019; Banerjee and Baral, 2020; Wu et al., 2020; Rücklé et al., 2020; Shi et al., 2020; Yamada et al., 2020; Xu et al., 2020; Guu et al., 2020).”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, methodology): “and unsupervised approaches for zero-shot transfer for community question answering (Rücklé et al., 2020b).”
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models (2020, methodology): “…efficient setup based on adapters (Rebuffi et al., 2017; Houlsby et al., 2019; Stickland and Murray, 2019; Pfeiffer et al., 2020a,b,c,d; Lauscher et al., 2020a; Rücklé et al., 2020a,b, inter alia): additional parameter sets that are fine-tuned while the original pretrained model is kept frozen.”