MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension

By Alon Talmor, Jonathan Berant. 2019

176 citations recorded by the source corpus.

A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generalization, as well as transfer to a target RC dataset. We analyze the factors that contribute to generalization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual representations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MultiQA, a BERT-based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “For example, Clark et al. (2019c) multiple-choice datasets, while Talmor and Berant (2019) focus their generalization study on extractive span prediction models.”
A Constructive Prediction of the Generalization Error Across Scales (2019, background): “Data scaling: It has long been recognized that more data improves performance, and various studies report such trends in both computer vision (e.g., Zhu et al., 2012; Sun et al., 2017) and language processing tasks (e.g., Banko & Brill, 2001; Talmor & Berant, 2019).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “While work exists that investigates the use of web content in KI-NLP, it usually relies on proprietary, black-box search engines to find documents relevant to the input queries and focuses on individual tasks (primarily ODQA) (Joshi et al., 2017; Bajaj et al., 2018; Talmor and Berant, 2019), or only”
The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “Given the plethora of question-answering datasets, Yogatama et al. (2019), Talmor & Berant (2019), and Sen & Saffari (2020) evaluate the extent to which models trained on SQuAD generalize to other question-answering datasets.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, background): “On the other hand, recurring findings reveal 18 that such models are brittle when provided with out-of-domain [1, 2, 3] or adversarial examples 19 [4, 5, 6, 7, 8, 9, 10], and make silly mistakes that contradict basic notions of human common sense.”
The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents (2019, background): “Talmor and Berant (2019) collect a set of 10 QA datasets and build M ULTI QA.”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “, 2018), following the MultiQA model (Talmor and Berant, 2019).”
Exploring and Predicting Transferability across NLP Tasks (2020, methodology): “…33 datasets in our study.7 We select these datasets by mostly following prior work: nine of the eleven CR tasks come from the GLUE benchmark (Wang et al., 2019b); all eleven QA tasks are from the MultiQA repository (Talmor and Berant, 2019b); and all eleven SL tasks were used by Liu et al. (2019a).”
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020, background): “Talmor and Berant (2019) perform a similar cross-task transfer study on reading comprehension datasets, ﬁnding similar positive transfer in most cases, with the biggest gains stemming from a combination of multiple QA datasets.”
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2021, background): “, 2019), while SQUAD was found to generalize well to QA tasks (Talmor and Berant, 2019).”
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks (2020, background): “A recent step towards robustness of verbal reasoning is training on a combination of different datasets (Talmor and Berant 2019).”
What Do Models Learn from Question Answering Datasets? (2020, background): “This finding confirms results found in previous work on different mixes of datasets (Talmor and Berant, 2019; Yogatama et al., 2019).”
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering (2019, background): “This confirms the observations from Talmor and Berant (2019) that multi-domain training improves robustness and generalization broadly, and suggests that a variety of question answering domains is significant across domains.”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “Data augmentation techniques have been extensively explored in RC, QA, and dialogue (Feng et al., 2021; Talmor and Berant, 2019; Khashabi et al., 2020; Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021).”
Syntactic Data Augmentation Increases Robustness to Inference Heuristics (2020, methodology): “This discrepancy can manifest as a sharp drop in accuracy when the model is applied to a different dataset that illustrates the same task (Talmor and Berant, 2019; Yogatama et al., 2019), or as excessive sensitivity to linguistically irrelevant perturbations of the input (Jia and Liang, 2017;…”
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge (2020, methodology): “Training is done by randomly mixing the datasets and performing standard multi-task training [18, 19].”
Look at the First Sentence: Position Bias in Question Answering (2020, methodology): “5 We choose ﬁrstly occurring answers for training extractive QA models, which is a common approach in weakly supervised setting (Joshi et al., 2017; Talmor and Berant, 2019).”