MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension

By Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen. 2019

335 citations recorded by the source corpus.

We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the rest were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “The MRQA shared task (Fisch et al., 2019) focuses on span-prediction datasets.”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “We also evaluate on five more datasets from the MRQA shared task (Fisch et al., 2019)4: NewsQA (Trischler et al.”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, neutral): “, 2016) and other question-answering datasets (Fisch et al., 2019).”
Longformer: The Long-Document Transformer (2020, methodology): “We use the full version of TriviaQA and HotpotQA, not the simplified versions in MRQA (Fisch et al., 2019).”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, methodology): “For question answering, we use the MRQA 2019 shared task on generalization (Fisch et al., 2019).”
The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “In a similar vein, Fisch et al. (2019) conduct a shared task competition that evaluates how well models trained on multiple datasets generalize to unseen datasets at test time.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, background): “On the other hand, recurring findings reveal 18 that such models are brittle when provided with out-of-domain [1, 2, 3] or adversarial examples 19 [4, 5, 6, 7, 8, 9, 10], and make silly mistakes that contradict basic notions of human common sense.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “Table 6 shows the breakdown of results on the 12 MRQA in-and out-of domain evaluation sets.”
Evaluating NLP Models via Contrast Sets (2020, background): “Generalization to new data distributions The MRQA shared task (Fisch et al., 2019) evaluates generalization to held-out datasets which require different types of reasoning (e.g., numerical reasoning, compositional questions) and come from different domains (e.g., biomedical, newswire, Wikipedia).”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, methodology): “We explore mixing datasets from different NLP benchmarks or families of tasks, including GLUE, SUPERGLUE, natural language inference (NLI), paraphrasing/semantic similarity, sentiment analysis, question answering (QA) on MRQA (Fisch et al., 2019), commonsense reasoning on RAINBOW (Lourie et al., 202”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “Furthermore, we evaluate the ﬁnal models on three existing robustness measures and achieve state-of-the-art results on AdversarialQA, improved learnability of various comprehension skills for CheckList, and improved domain gener-alisation for the suite of MRQA tasks.”
Entity-Based Knowledge Conflicts in Question Answering (2021, methodology): “ally sourced as passages of text from the retriever (Fisch et al., 2019).”
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (2020, background): “MRQA shared task (Fisch et al. 2019) and Open Reading Benchmark 2 offer a curated collection of extractive question answering datasets with different domains.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “We only use the questions from MRQA as the new questions in the experiments of NQ.”
Few-Shot Question Answering by Pretraining Span Selection (2021, methodology): “In this work, we follow the assumptions in the recent MRQA 2019 shared task (Fisch et al., 2019) and focus on questions whose answer is a single span.”
What Do Models Learn from Question Answering Datasets? (2020, methodology): “The MRQA 2019 shared task (Fisch et al., 2019) evaluated transferability across multiple datasets, and Khashabi et al. (2020) proposed a method to train one model on 17 different QA datasets.”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “U NIFIED -IO is trained on several QA datasets including SQuAD 2.0 (Rajpurkar et al., 2016), other training datasets from the MRQA (Fisch et al., 2019) shared tasks (Trischler et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019), QA datasets from…”
A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering (2020, methodology): “Our model for reading comprehension was also pretrained using MultiQA dataset (Fisch et al., 2019)”
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking (2021, background): “…[62] recast as NLI [53] QA SQuAD dev [57], AdversarialQA [1] (Dynabench QA round 1) AdversarialQA dev, the 12 dev sets from MRQA shared tasks [18] Sentiment SST3 [66], DynaSent [55] Respective dev sets; Amazon Reviews [78] test and dev (10k subsample), Yelp Reviews [78] test and dev (10k…”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, methodology): “models to evaluate multiple skills could aid multitask or multi-metric leaderboards like MRQA (Fisch et al., 2019) and Dynaboard (Ma et al.”