Question Answering Infused Pre-training of General-Purpose Contextualized Representations

By Robin Jia, Michael Lewis, Luke S. Zettlemoyer. 2021

30 citations recorded by the source corpus.

We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations, motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context. To this end, we train a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs. By encoding QA-relevant information, the bi-encoder’s token-level representations are useful for non-QA downstream tasks without extensive (or in some cases, any) fine-tuning. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection on four datasets, few-shot named entity recognition on two datasets, and zero-shot sentiment analysis on three datasets.

Subfields: Artificial Intelligence, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Sebastian Riedel
3. Yi Tay
4. Wen-tau Yih
5. Mohit Bansal
6. Caiming Xiong
7. Hannaneh Hajishirzi
8. Jianfeng Gao

In Natural Language Processing

1. Mohit Bansal
2. Caiming Xiong
3. Luke S. Zettlemoyer
4. Sebastian Riedel
5. Wen-tau Yih
6. Sebastian Ruder
7. Yi Tay
8. Graham Neubig

In Prompting & Reasoning

1. Ramakanth Pasunuru
2. Ves Stoyanov
3. Wen-tau Yih
4. Yi Tay
5. Patrick Lewis
6. Jiawei Han
7. Yu Meng
8. Donald Metzler

Citations in the corpus

• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “Efforts have also been made to tailor pre-training objectives to specific tasks, e.g., question answering (Ram et al., 2021; Jia et al., 2021), dialogue (Li et al., 2020), and span selection tasks (Joshi et al., 2020).”
• Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “Ma et al. (2021) and Jia et al. (2021) also investigate training on generated QA pairs, but the former only considers application to domain transfer and the latter to other NLP tasks.”
• QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (2021, methodology): “We additionally experiment with two models trained from RoBERTa-large checkpoints, one trained from the original checkpoint, LERC (RoBERTa), and one initialized from Jia et al. (2021), which we call LERC (QuIP), for the task of jointly encoding passages and answers with question-infused pretraining.”
• Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, background): “, 2021b), question-infused pretraining (Jia et al., 2021), and self-training (Du et al.”
• Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators (2022, background): “The advantage of manually constructed pretraining tasks is more often observed in an application-specific manner, where prior knowledge about the target scenario is introduced (Guu et al., 2020; Roberts et al., 2020; Jia et al., 2021; Lu et al., 2021).”
• CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training (2021, background): “For example, Lee et al. (2019) pro-165 pose the Inverse Cloze Task (ICT), Chang et al.166 (2020) introduce Body First Selection (BFS) and167 Wiki Link Prediction (WLP) and Jia et al. (2021)168 describe a novel Question Answering Infused Pre-169 training (QUIP) task.”