Quasar: Datasets for Question Answering by Search and Reading

By Bhuwan Dhingra, Kathryn Mazaitis, William W. Cohen. 2017

194 citations recorded by the source corpus.

We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S dataset consists of 37000 cloze-style (fill-in-the-gap) queries constructed from definitions of software entity tags on the popular website Stack Overflow. The posts and comments on the website serve as the background corpus for answering the cloze questions. The Quasar-T dataset consists of 43000 open-domain trivia questions and their answers obtained from various internet sources. ClueWeb09 serves as the background corpus for extracting these answers. We pose these datasets as a challenge for two related subtasks of factoid Question Answering: (1) searching for relevant pieces of text that include the correct answer to a query, and (2) reading the retrieved text to answer the query. We also describe a retrieval system for extracting relevant sentences and documents from the corpus given a query, and include these in the release for researchers wishing to only focus on (2). We evaluate several baselines on both datasets, ranging from simple heuristics to powerful neural models, and show that these lag behind human performance by 16.4% and 32.1% for Quasar-S and -T respectively. The datasets are available at this https URL .

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Simple and Effective Multi-Paragraph Reading Comprehension (2017, neutral): “Another dataset of trivia questions, Quasar-T (Dhingra et al., 2017), was introduced recently that uses ClueWeb09 (Callan et al., 2009) as its source for documents.”
Improving Question Answering with External Knowledge (2019, neutral): “In contrast to other types of QA scenarios (Nguyen et al., 2016; Dhingra et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Kwiatkowski et al., 2019), in this setting: (1) the reference corpus does not reliably contain text spans from which the answers can be drawn, and (2) it does not provide suf”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, background): “In work that is more closely related to our approach, the reader is learned entirely from weak supervision (Joshi et al., 2017; Dhingra et al., 2017; Dunn et al., 2017).”
Passage Re-ranking with BERT (2019, methodology): “We have seen rapid progress in machine reading compression in recent years with the introduction of large-scale datasets, such as SQuAD (Rajpurkar et al., 2016), MS MARCO (Nguyen et al., 2016), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and QUASAR-T (Dhingra et al., 2017), and the ”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, methodology): “By contrast, in the open-domain setting, the RC models are usually trained under distant supervision (Chen et al., 2017; Dhingra et al., 2017b; Joshi et al., 2017).”
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (2019, neutral): “(3) Quasar-T (Dhingra et al., 2017) and (4) SearchQA (Dunn et al.”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, methodology): “(d) QUASAR-T (Dhingra et al., 2017) – consists of 43K open-domain questions where the paragraphs are obtained from the ClueWeb data source.”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “Quasar-T1 (Dhingra et al., 2017b) consists of 43, 000 open-domain trivia question, and their answers are extracted from ClueWeb09 data source, and the paragraphs are obtained by retrieving 50 sentences for each question from the ClueWeb09 data source using LUCENE.”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, neutral): “, 2017), QuasarT (Dhingra et al., 2017), WebQuestions (Berant”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “, Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al.”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, background): “Recently, researchers have developed large-scale QA datasets, which requires answering the question over a large set of documents in a closed (Joshi et al., 2017) or open-domain (Dunn et al., 2017; Berant et al., 2013; Chen et al., 2017; Dhingra et al., 2017).”
R3: Reinforced Reader-Ranker for Open-Domain Question Answering (2017, methodology): “We also compare our model R 3 with two internal baselines: Single Reader (SR) This model is trained in the same way as Chen et al. 2017a and Dhingra et al. 2017b.”
Combining Fact Extraction and Verification with Neural Semantic Matching Networks (2018, background): “Following their work, Dhingra, Mazaitis, and Cohen (2017) introduced new benchmarks, Wang et al. (2018) extended the framework by adding feedback signals from comprehension to the upstream document retriever, and Kratzwald and Feuerriegel (2018) proposed to adaptively adjust the number of retrieved…”
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (2019, background): “• Quasar-T (Dhingra et al., 2017) is another dataset that includes trivia questions.”
Learning to Generate Questions by Learning to Recover Answer-containing Sentences (2021, methodology): “To show that our generated data is useful for other MRC datasets, we ﬁne-tune and test the MRC model on QUASAR-T (Dhingra et al., 2017), which is another large-scale MRC dataset, after training on the synthetic data generated from 3 https://github.com/google-research/electra”
Evidence Extraction for Machine Reading Comprehension with Deep Probabilistic Logic (2018, methodology): “Wang et al. (2016); Dhingra et al. (2017c, 2018) show that the trained model doesn’t learn too much information and the traditional linguistics information is still valuable in end-to-end machine comprehension model.”
A Multi-Stage Memory Augmented Neural Network for Machine Reading Comprehension (2018, background): “The QUASAR-T dataset consists of factoid question-answer pairs and a corresponding large background corpus (Callan et al., 2009) to comprehend.”
Open-Domain Question Answering (2020, background): “We will then discuss modern datasets proposed for open-domain QA (Voorhees et al., 1999; Berant et al., 2013; Rajpurkar et al., 2016; Joshi et al., 2017; Dhingra et al., 2017; Dunn et al., 2017; Kwiatkowski et al., 2019), as well as common evaluation metrics and benchmarks.”
Retrieve, Read, Rerank: Towards End-to-End Multi-Document Reading Comprehension (2019, background): “Quasar-T (Dhingra et al., 2017) and SearchQA (Dunn et al., 2017), however, pair each question-answer pair with a set of web page snippets that are more analogous to paragraphs.”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, background): “Recently, multiple datasets have been proposed for multi-hop QA, in which questions can only be answered when considering information from multiple sentences and/or documents (Clark et al., 2018; Khashabi et al., 2018a; Yang et al., 2018; Welbl et al., 2018; Mihaylov et al., 2018; Bauer et al., 2018”