Span Selection Pre-training for Question Answering

By Michael R. Glass, Alfio Massimiliano Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, G P Shrivatsa Bhargav, Dinesh Garg, Avirup Sil. 2019

76 citations recorded by the source corpus.

BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pretrained on two auxiliary tasks: Masked Language Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memorization to understanding. Span Selection PreTraining (SSPT) poses cloze-like training instances, but rather than draw the answer from the model’s parameters, it is selected from a relevant passage. We find significant and consistent improvements over both BERT-BASE and BERT-LARGE on multiple Machine Reading Comprehension (MRC) datasets. Specifically, our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming BERT-LARGE by 3 F1 points on short answer prediction. We also show significant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effective when training data is limited, improving the learning curve by a large amount.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, neutral): “Glass et al. (2020) for QA.”
Few-Shot Question Answering by Pretraining Span Selection (2021, methodology): “Both Lewis et al. (2019) and Glass et al. (2020) use the traditional NLP pipeline to select noun phrases and named entities in Wikipedia paragraphs as potential answers, which are then masked from the context to create pseudo-questions.”
Frustratingly Easy Natural Question Answering (2019, methodology): “BERT with Span Selection Pre-Training (SSPT) uses an unsupervised auxiliary QA specific task proposed by Glass et al. (2019) to further train the BERT model.”
RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020, methodology): “Glass et al. (2019) propose a span selection method for BERT pre-training (SSPT).”
Pruning a BERT-based Question Answering Model (2019, methodology): “(Glass et al., 2019) (This model was initialized from bert-large-uncased .)”
ReasonBERT: Pre-trained to Reason with Distant Supervision (2021, methodology): “Compared with pre-training methods such as SSPT and Splinter, ReasonBERT also shows superior performance and obtains the best results on average.”
Is Graph Structure Necessary for Multi-hop Question Answering? (2020, unknown): “Joint EM Joint F1 Baseline (Yang et al., 2018) 10.83 40.16 QFE (Nishida et al., 2019) 34.63 59.61 DFGN (Qiu et al., 2019) 33.62 59.82 TAP2 (Glass et al., 2019) 39.77 69.12 HGN (Fang et al., 2019) 43.57 71.03 SAE (Tu et al., 2019a) 45 eral extra modules in the graph fusion block, including…”