Few-Shot Question Answering by Pretraining Span Selection

By Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, Omer Levy. 2021

104 citations recorded by the source corpus.

In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available, and observe that standard models perform poorly, highlighting the discrepancy between current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question answering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD with only 128 training examples), while maintaining competitive performance in the high-resource setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, methodology): “Our pretraining is based on the notion of recurring spans (Ram et al., 2021) within a document: given two paragraphs with the same recurring span, we construct a query from one of the paragraphs, while the other is taken as the target for retrieval (Figure 2).”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “Efforts have also been made to tailor pre-training objectives to specific tasks, e.g., question answering (Ram et al., 2021; Jia et al., 2021), dialogue (Li et al., 2020), and span selection tasks (Joshi et al., 2020).”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “Other work on pre-training focuses on tailored masking schemes such as recurrent span masking (Ram et al., 2021) for question answering and salient sentence masking for summarization (Zhang et al., 2020).”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “Up to now, the FSL and fine-tuning have been widely used in tasks like plant disease and insect pest identification [80], road detection [81], and automatic question and answer [82].”
ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, methodology): “Compared with pre-training methods such as SSPT and Splinter, ReasonBERT also shows superior performance and obtains the best results on average.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, background): “Ram et al. (2021) propose a span extraction pre-training objective that enables few-shot QA.”
How Optimal is Greedy Decoding for Extractive Question Answering? (2021, methodology): “We compare greedy decoding with exactextract on the recently-proposed few-shot question answering benchmark (Ram et al., 2021), which contains subsampled training sets of 16 to 1024 examples from 8 different datasets.”