SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine

By Matthew Dunn, Levent Sagun, Mike Higgins, V. U. Güney, Volkan Cirik, Kyunghyun Cho. 2017

496 citations recorded by the source corpus.

We publicly release a new large-scale dataset, called SearchQA, for machine comprehension, or question-answering. Unlike recently released datasets, such as DeepMind CNN/DailyMail and SQuAD, the proposed SearchQA was constructed to reflect a full pipeline of general question-answering. That is, we start not from an existing article and generate a question-answer pair, but start from an existing question-answer pair, crawled from J! Archive, and augment it with text snippets retrieved by Google. Following this approach, we built SearchQA, which consists of more than 140k question-answer pairs with each pair having 49.6 snippets on average. Each question-answer-context tuple of the SearchQA comes with additional meta-data such as the snippet's URL, which we believe will be valuable resources for future research. We conduct human evaluation as well as test two baseline methods, one simple word selection and the other deep learning based, on the SearchQA. We show that there is a meaningful gap between the human and machine performances. This suggests that the proposed dataset could well serve as a benchmark for question-answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “(Dunn et al., 2017) where Jeopardy! questions are paired with search engine snippets, the WikiQA dataset (Yang et al.”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “We use the raw Jeopardy data and splits from SearchQA [10], consisting of 97,391 training, 13,713 development, and 26,848 test datapoints.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
CoQA: A Conversational Question Answering Challenge (2018, background): “Elgohary et al. (2018) proposed a sequential question answering dataset collected from Quiz Bowl tournaments, where a sequence contains multiple related questions.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, neutral): “We train on 140 GB of text drawing from a wide variety of domains: Wikipedia (En, De, Es, Fr), Project Gutenberg 1 , submissions from 45 subreddits, OpenWebText 2 , a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McA”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “…similarity tasks: MRPC (Dolan and Brockett, 2005), STS-B (Cer et al., 2017), QQP,3 and 4 natural language inference tasks: MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007) and WNLI (Levesque et al., 2011).”
Improving Question Answering with External Knowledge (2019, neutral): “In contrast to other types of QA scenarios (Nguyen et al., 2016; Dhingra et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Kwiatkowski et al., 2019), in this setting: (1) the reference corpus does not reliably contain text spans from which the answers can be drawn, and (2) it does not provide suf”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “ trivia questions that are carefully crafted by someone who already knows the answer such that exactly one answer is correct such as TriviaQA and Quizbowl/Jeopoardy! questions (Ferrucci et al., 2010; Dunn et al., 2017; Joshi et al., 2017; Peskov et al., 2019); this information-verifying paradigm als”
ELI5: Long Form Question Answering (2019, methodology): “, 2017), SearchQA (Dunn et al., 2017), and QuAC (Choi et al.”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, background): “In work that is more closely related to our approach, the reader is learned entirely from weak supervision (Joshi et al., 2017; Dhingra et al., 2017; Dunn et al., 2017).”
Passage Re-ranking with BERT (2019, methodology): “We have seen rapid progress in machine reading compression in recent years with the introduction of large-scale datasets, such as SQuAD (Rajpurkar et al., 2016), MS MARCO (Nguyen et al., 2016), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and QUASAR-T (Dhingra et al., 2017), and the ”
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering (2017, methodology): “, 2017b), SearchQA (Dunn et al., 2017) and TriviaQA (Joshi et al.”
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (2019, background): “4) SearchQA (Dunn et al., 2017) are leveraged with the ofﬁcial split.”
Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering (2019, background): “Open-domain QA has been popularized by the Trec-8 task (Voorhees et al., 1999) and has recently gained signiﬁcant traction due to the introduction of various datasets (Dunn et al., 2017; Dhingra et al., 2017; Joshi et al., 2017).”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “SearchQA 2 (Dunn et al., 2017) is a large-scale open domain question answering dataset, which consists of question-answer pairs crawled from J!”
AmbigQA: Answering Ambiguous Open-domain Questions (2020, background): “Existing benchmarks include various kinds of questions, from open-ended information-seeking (Be-rant et al., 2013; Kwiatkowski et al., 2019; Clark et al., 2019) to more specialized trivia/quiz (Joshi et al., 2017; Dunn et al., 2017).”
The NarrativeQA Reading Comprehension Challenge (2017, background): “SearchQA (Dunn et al., 2017) is a recent dataset in which the context for each question is a set of documents retrieved by a search engine using the question as the query.”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018, background): “, 2017) and SearchQA (Dunn et al., 2017) create a more”
The Effect of Natural Distribution Shift on Question Answering Models (2020, background): “Beyond SQuAD, a number of works have proposed datasets for question answering (Richardson et al., 2013; Berant et al., 2014; Joshi et al., 2017; Trischler et al., 2017; Dunn et al., 2017; Yang et al., 2018; Kwiatkowski et al., 2019).”