Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks

By Anna Rogers, Olga Kovaleva, Matthew T. Downey, Anna Rumshisky. 2020

141 citations recorded by the source corpus.

The recent explosion in question answering research produced a wealth of both factoid reading comprehension (RC) and commonsense reasoning datasets. Combining them presents a different kind of task: deciding not simply whether information is present in the text, but also whether a confident guess could be made for the missing information. We present QuAIL, the first RC dataset to combine text-based, world knowledge and unanswerable questions, and to provide question type annotation that would enable diagnostics of the reasoning strategies by a given QA system. QuAIL contains 15K multi-choice questions for 800 texts in 4 domains. Crucially, it offers both general and text-specific questions, unlikely to be found in pretraining data. We show that QuAIL poses substantial challenges to the current state-of-the-art systems, with a 30% drop in accuracy compared to the most similar existing dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

UL2: Unifying Language Learning Paradigms (2022, positive): “Concretely, we use RACE (Reading comprehension) (Lai et al., 2017), QASC (Khot et al., 2020), OpenBookQA (Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., ”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…and Uthus 2020 proto_qa other Boratko et al. 2020 qa_srl other He et al. 2015 qasc qa/multiple-choice qa Khot et al. 2020 quail qa/multiple-choice qa Rogers et al. 2020 quarel qa/multiple-choice qa Tafjord et al. 2019a quartz-no_knowledge qa/multiple-choice qa Tafjord et al. 2019b…”
When BERT Plays the Lottery, All Tickets Are Winning (2020, background): “One of the most famous Transformers is BERT (Devlin et al., 2019), which became a must-have baseline and inspired dozens of analysis studies (Rogers et al., 2020b).”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “Looking further afield from these question types above, which are less frequent in NQ but nevertheless attested there, it becomes clear that QA writ large is much broader than even a dataset of NQ’s scale suggests (Rogers et al., 2020).”
What to Pre-Train on? Efficient Intermediate Task Selection (2021, background): “Additionally, we include other recent challenging tasks that fall under the four deﬁned task types (e.g. Bhagavatula et al. (2020); Rogers et al. (2020)) and tasks that extend the range of included dataset sizes and task domains.”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, methodology): “Rogers et al. (2020) constrain the task design for QuAIL and require workers to write questions of speciﬁc types, like those targeting temporal reasoning.”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, background): “In this setup: We evaluate the models on the following unseen AdversarialQA, ReCoRD, RACE-C, HeadQA, MMMLU, ReClor, Quail, OneStopQA, MCScript, MCScript 2.0, CosmosQA, DREAM, ProcessBank, PROST, StrategyQA, PubmedQA, QAConv, TweetQA.”
Unifying Language Learning Paradigms (2022, unknown): “Concretely, we use RACE (Reading comprehension) (Lai et al., 2017), QASC (Khot et al., 2020), OpenBookQA (Mihaylov et al., 2018), TweetQA (Xiong et al., 2019), QuAIL (Rogers et al., 2020), IMDB (Maas et al., 2011), Agnews (Zhang et al., 2015), DocNLI (Yin et al., 2021), Adversarial NLI (Nie et al., ”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, background): “Several other benchmarks in this vein also show bad performance: ReClor [Yu et al., 2020], QuAIL [Rogers et al., 2020a], ConTRoL [Liu et al., 2020a], Strate-6 gyQA [Geva et al., 2021], AR-LSAT [Zhong et al., 2021], and CLUTRR [Sinha et al., 2019].”
Changing the World by Changing the Data (2021, methodology): “The paradigm of pre-training followed by ﬁne-tuning on downstream tasks was popularized by BERT (Devlin et al., 2019), and is actively developed (Rogers et al., 2020b).”
TIES-Merging: Resolving Interference When Merging Models (2023, methodology): “Specifically, we report the average performance over the following tasks and datasets: Cosmos QA [27], Social IQA [67], and QuAIL [62] for question answering; WiC [53] for word sense disambiguation; and COPA [61], and H-SWAG [88] for sentence completion.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “There is also work that proposes datasets with examples spanning multiple domains (Rogers et al., 2020; Reddy et al., 2019; Hendrycks et al., 2021).”
Comparing Test Sets with Item Response Theory (2021, methodology): “0 and QuAIL, we analyze the context length, the answerability of a question, and the lexical overlap between context and questions.”