Proceedings of the 2nd Workshop on Machine Reading for Question Answering

By Dianhai Yu, Heng Ji, Dong Yu. 2019

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems.1 In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training, and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.

Subfields: Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Question Answering & Reading Comprehension

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “Unlike Fisch et al. (2019); Yogatama et al. (2019), we measure OOD generalization by considering simple and natural distribution shifts, and we also evaluate more than question answering.”
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts (2020, background): “• Information Integration over Paragraphs Multi-step reasoning (Liu et al. 2020b; Wang et al. 2019; Welbl, Stenetorp, and Riedel 2018) is the ability to retrieve and combine information from multiple paragraphs or multiple documents.”
What Question Answering can Learn from Trivia Nerds (2019, background): “Paraphrasing a participant of the 2019 MRQA workshop (Fisch et al., 2019), a system better than the average human at brain surgery does not imply superhuman performance in brain surgery.”