WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia

By Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, David Berthelot. 2016

146 citations recorded by the source corpus.

We present WIKIREADING, a large-scale natural language understanding task and publicly-available dataset with 18 million instances. The task is to predict textual values from the structured knowledge base Wikidata by reading the text of the corresponding Wikipedia articles. The task contains a rich variety of challenging classification and extraction sub-tasks, making it well-suited for end-to-end models such as deep neural networks (DNNs). We compare various state-of-the-art DNNbased architectures for document classification, information extraction, and question answering. We find that models supporting a rich answer space, such as word or character sequences, perform best. Our best-performing model, a word-level sequence to sequence model with a mechanism to copy out-of-vocabulary words, obtains an accuracy of 71.8%.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, background): “Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “Additional datasets including SQuAD (Ra-jpurkar et al., 2016), WikiReading (Hewlett et al., 2016), MS Marco (Nguyen et al., 2016) and Triv-iaQA (Joshi et al., 2017) provided more realistic questions.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
Generating Wikipedia by Summarizing Long Sequences (2018, background): “There is a rich body of work incorporating Wikipedia for machine learning tasks, including question-answering (Hewlett et al. (2016), Rajpurkar et al. (2016)) and information extraction (Lehmann et al., 2015), and text generation from structured data (Lebret et al., 2016).”
Reading Wikipedia to Answer Open-Domain Questions (2017, background): “That subﬁeld has made considerable progress recently thanks to new deep learning architectures like attention-based and memory-1 augmented neural networks (Bahdanau et al., 2015; Weston et al., 2015; Graves et al., 2014) and release of new training and evaluation datasets like QuizBowl (Iyyer et al.”
Coarse-to-Fine Question Answering for Long Documents (2016, methodology): “1 The resulting W IKIREADING L ONG dataset contains 1.97M examples, where the answer appears in 50.4% of the examples, and appears in the ﬁrst sentence only 31% of the time.”
The NarrativeQA Reading Comprehension Challenge (2017, methodology): “ As we have multiple questions per summary/story, this allows us to consider answer selection (from among the 30) as a simpler version of the QA than answer generation from scratch. Answer selection (Hewlett et al., 2016) and multiple-choice question answering (Richardson et al., 2013; Hill et al., ”
Zero-Shot Relation Extraction via Reading Comprehension (2017, methodology): “Slot-Filling Data We use the WikiReading dataset (Hewlett et al., 2016) to collect labeled”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “For both datasets we draw upon existing Knowledge Bases (KBs), W IKIDATA and D RUG B ANK , as ground truth, utilising distant super-vision (Mintz et al., 2009) to induce the data – similar to Hewlett et al. (2016) and Joshi et al. (2017).”
Neural Approaches to Conversational AI (2018, positive): “ recent big progress on MRC is largely due to the availability of a multitude of large-scale datasets that the research community has created over various text sources such as Wikipedia (WikiReading (Hewlett et al., 2016), SQuAD (Rajpurkar et al., 2016), WikiHop (Welbl et al., 2017), DRCD (Shao et a”
The Web as a Knowledge-Base for Answering Complex Questions (2018, background): “While many QA datasets have been developed recently (Yang et al., 2015; Rajpurkar et al., 2016; Hewlett et al., 2016; Nguyen et al., 2016; Onishi et al., 2016; Hill et al., 2015; Welbl et al., 2017), they lack a focus on the importance of question decomposition.”
Machine Comprehension Using Match-LSTM and Answer Pointer (2016, methodology): “Besides the datasets above, there are also a few other datasets created for machine comprehension, such as WikiReading dataset (Hewlett et al., 2016) and bAbI dataset (Weston et al., 2016), but they are quite different from the datasets above in nature.”
Crowdsourcing Multiple Choice Science Questions (2017, methodology): “…entailment, reading comprehension and Question Answering (QA) (Bowman et al., 2015; Hermann et al., 2015; Rajpurkar et al., 2016; Hill et al., 2015; Hewlett et al., 2016; Nguyen et al., 2016) has allowed for advances on these tasks, particularly with neural models (Kadlec et al., 2016; Dhingra et…”
What Makes Reading Comprehension Questions Easier? (2018, background): “Other datasets that are not covered in our study, but can be analyzed using the same method, include: QA4MRE (Sutcliffe et al., 2013), CNN/Daily Mail (Hermann et al., 2015), Children’s Book Test (Hill et al., 2016), bAbI (Weston et al., 2015), WikiReading (Hewlett et al., 2016), LAMBADA (Paperno et ”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “The authors use the WikiReading dataset (Hewlett et al., 2016) for the underlying slot-filling task.”
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis (2019, methodology): “, Wikipedia (WikiReading (Hewlett et al., 2016), SQuAD (Rajpurkar et al.”
Embracing data abundance: BookTest Dataset for Reading Comprehension (2016, methodology): “In the WikiReading (Hewlett et al., 2016) dataset the context document is formed from a Wikipedia article and the question-answer pair is taken from the corresponding WikiData page.”