Read + Verify: Machine Reading Comprehension with Unanswerable Questions

By Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, Ming Zhou. 2018

164 citations recorded by the source corpus.

Machine reading comprehension with unanswerable questions aims to abstain from answering when no answer can be inferred. In addition to extract answers, previous works usually predict an additional “no-answer” probability to detect unanswerable cases. However, they fail to validate the answerability of the question by verifying the legitimacy of the predicted answer. To address this problem, we propose a novel read-then-verify system, which not only utilizes a neural reader to extract candidate answers and produce no-answer probabilities, but also leverages an answer verifier to decide whether the predicted answer is entailed by the input snippets. Moreover, we introduce two auxiliary losses to help the reader better handle answer extraction as well as no-answer detection, and investigate three different architectures for the answer verifier. Our experiments on the SQuAD 2.0 dataset show that our system obtains a score of 74.2 F1 on test set, achieving state-of-the-art results at the time of submission (Aug. 28th, 2018).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “We believe that the performance of the U-Net can be boosted with an additional post-processing step to verify answers using approaches such as (Hu et al. 2018).”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Following previous work (Hu et al., 2018), we also try building entailment-like training data from SQuAD 2.0 (Rajpurkar et al., 2018).”
Improving Question Answering with External Knowledge (2019, methodology): “, 2019) on QA tasks (Radford et al., 2018; Devlin et al., 2019; Hu et al., 2019; Yang et al., 2019).”
RECONSIDER: Re-Ranking using Span-Focused Cross-Attention for Open Domain Question Answering (2020, methodology): “Hu et al. (2019) use answer veriﬁ-cation to predict the unanswerability of a question-passage pair for traditional MRC tasks.”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “Research on unanswerable questions has attracted a lot of attention in the development of MRC over the past few years [20], [22], [125], [141], [147], [148].”
Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension (2019, background): “By only extracting an answer without verifying it, the models may predict a wrong answer and are unable to recover from such mistakes (Hu et al., 2019).”
Cross-Lingual Machine Reading Comprehension (2019, methodology): “…SQuAD (Rajpurkar et al., 2016)), and we have seen a great progress on re-lated neural network approaches (Wang and Jiang, 2016; Seo et al., 2016; Xiong et al., 2016; Cui et al., 2017; Hu et al., 2019), especially those were built on pre-trained language models, such as BERT (Devlin et al., 2019).”
Multi-style Generative Reading Comprehension (2019, unknown): “(2018); Indurthi et al. (2018); Hu et al.”
Can NLI Models Verify QA Systems' Predictions? (2021, background): “Another line of work to make models more robust is by introducing answer verification (Hu et al., 2019; Kamath et al., 2020; Wang et al., 2020) as a final step for question answering model.”
Retrospective Reader for Machine Reading Comprehension (2020, background): “Meanwhile, little attention is paid to the decoder side 1 of MRC models [ Hu et al. , 2019; Back et al. , 2020 ] , though it has been shown that better decoder or better manner of using encoder still has a signiﬁcant impact on MRC performance, no matter how strong the encoder (i.e., the adopted…”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, background): “, (Hu et al., 2019)) on addressing unanswerable questions in extractive machine reading comprehension tasks such as SQuAD (Rajpurkar et al.”