Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision
By Ning Bian, Xianpei Han, Bo Chen, Hongyu Lin, Ben He, Le Sun. 2021
5 citations recorded by the source corpus.
Despite recent success in machine reading comprehension (MRC), learning high-quality MRC models still requires large-scale labeled training data, even using strong pre-trained language models (PLMs). The pre-training tasks for PLMs are not question-answering or MRC-based tasks, making existing PLMs unable to be directly used for unsupervised MRC. Specifically, MRC aims to spot an accurate answer span from the given document, but PLMs focus on token filling in sentences. In this paper, we propose a new framework for unsupervised MRC. Firstly, we propose to learn to spot answer spans in documents via self-supervised learning, by designing a self-supervision pretext task for MRC - Spotting-MLM. Solving this task requires capturing deep interactions between sentences in documents. Secondly, we apply a simple sentence rewriting strategy in the inference stage to alleviate the expression mismatch between questions and documents. Experiments show that our method achieves a new state-of-the-art performance for unsupervised MRC.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jonathan Berant
- Omer Levy
- Ori Ram
- Amir Globerson
- Percy Liang
- Luke S. Zettlemoyer
- Mohit Bansal
- Yoav Goldberg
In Computer Vision and Pattern Recognition
- Omer Levy
- Jonathan Berant
- Ori Ram
- Amir Globerson
- Gal Shachaf
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
In Natural Language Processing
- Jonathan Berant
- Amir Globerson
- Omer Levy
- Ori Ram
- Percy Liang
- Luke S. Zettlemoyer
- Yoav Goldberg
- Mohit Bansal
In Representation & Self-Supervised Learning
Citations in the corpus
- Learning to Retrieve Passages without Supervision (2021, background): “…spans for self-supervised pretraining has previously been considered for numerous tasks, e.g. coreference resolution and coreferential reasoning (Kocijan et al., 2019; Varkel and Globerson, 2020; Ye et al., 2020) and question answering (Ram et al., 2021; Bian et al., 2021; Castel et al., 2021).”