WikiQA: A Challenge Dataset for Open-Domain Question Answering

By Yi Yang, Wen-tau Yih, Christopher Meek. 2015

1,019 citations recorded by the source corpus.

We describe the WIKIQA dataset, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering. Most previous work on answer sentence selection focuses on a dataset created using the TREC-QA data, which includes editor-generated questions and candidate answer sentences selected by matching content words in the question. WIKIQA is constructed using a more natural process and is more than an order of magnitude larger than the previous dataset. In addition, the WIKIQA dataset also includes questions for which there are no correct sentences, enabling researchers to work on answer triggering, a critical component in any QA system. We compare several systems on the task of answer sentence selection on both datasets and also describe the performance of a system on the problem of answer triggering using the WIKIQA dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Evaluation & Benchmarks

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “, 2017) where Jeopardy! questions are paired with search engine snippets, the WikiQA dataset (Yang et al., 2015) for answer sentence selection, and the Chinese language WebQA (Li et al.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, methodology): “WikiQA (Yang et al., 2015) query logs IR, sentence selection 3047”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, neutral): “WikiQA (Yang et al., 2015) pairs questions from Bing query logs with sentences from Wikipedia.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “, 2016), WikiQA (Yang et al., 2015), and TriviaQA (Joshi et al.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of information-seeking questions such as WikiQA (Yang et al., 2015), MS MARCO (Nguyen et al., 2016), QuAC Pronounced tie dye Q. A. —like the colorful t-shirt.”
Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “Our model takes a hierarchical approach (see Figure 1), where ﬁrst a fast model is used to select a few sentences from the document that are relevant for answering the question (Yu et al., 2014; Yang et al., 2016a).”
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016, background): “WikiQA is another set which includes 3047 questions [32].”
Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index (2019, background): “2000s, some of the most notable datasets being TrecQA (Voorhees and Tice, 2000) and WikiQA (Yang et al., 2015).”
Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering (2019, background): “, 2013), WikiQA (Yang et al., 2015), and MSMARCO (Bajaj et al.”
Learning Recurrent Span Representations for Extractive Question Answering (2016, methodology): “Machine learned extractive question answering systems, such as the one presented here, learn a predictor function f ( q , p ) → a from a training dataset of (cid:104) q , p , a (cid:105) triples.”
Key-Value Memory Networks for Directly Reading Documents (2016, methodology): “Note that this is much larger than most existing datasets; for example, the WIKIQA dataset (Yang et al., 2015) for which we also conduct experiments in Sec.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Monolingual QA Data There is a great variety of English QA data, popularized by MCTest (Richardson, 2013), CNN/Daily Mail (Hermann et al., 2015) CBT (Hill et al., 2016), and Wik-iQA (Yang et al., 2015) amongst others.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Historically, many of the question answering datasets have only thousands of question answering pairs, such as WebQues-tions (Berant et al., 2013), MCTest (Richardson et al., 2013), WikiQA (Yang et al., 2015), and TREC-QA (Voorhees and Tice, 2000).”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Those that are labeled by humans are always in high quality (Richardson et al., 2013; Berant et al., 2014; Yang et al., 2015), but are too small for training modern data-intensive models.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “WIKIPEDIA has thus been used for a wealth of research to build datasets posing queries about a single sentence (Morales et al., 2016; Levy et al., 2017) or article (Yang et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016).”
The Web as a Knowledge-Base for Answering Complex Questions (2018, background): “While many QA datasets have been developed recently (Yang et al., 2015; Rajpurkar et al., 2016; Hewlett et al., 2016; Nguyen et al., 2016; Onishi et al., 2016; Hill et al., 2015; Welbl et al., 2017), they lack a focus on the importance of question decomposition.”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, neutral): “…2019 web_questions qa/closed-book qa Berant et al. 2013 wiki_auto cls/other Jiang et al. 2020 wiki_bio cg/other Lebret et al. 2016 wiki_qa cls/other Yang et al. 2015 wiki_split cg/other Botha et al. 2018 wikisql cg/other Zhong et al. 2017 wino_grande qa/multiple-choice qa Sakaguchi et al. 2020…”