Embracing data abundance: BookTest Dataset for Reading Comprehension

By Ondrej Bajgar, Rudolf Kadlec, Jan Kleindienst. 2016

69 citations recorded by the source corpus.

There is a practically unlimited amount of natural language data available. Still, recent work in text comprehension has focused on datasets which are small relative to current computing possibilities. This article is making a case for the community to move to larger data and as a step in that direction it is proposing the BookTest, a new dataset similar to the popular Children's Book Test (CBT), however more than 60 times larger. We show that training on the new data improves the accuracy of our Attention-Sum Reader model on the original CBT test data by a much larger margin than many recent attempts to improve the model architecture. On one version of the dataset our ensemble even exceeds the human baseline provided by Facebook. We then show in our own human study that there is still space for further improvement.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning (2017, background): “However, recent work has shown some success in using unsupervised pretraining with unlabeled data to both improve the performance of these methods and extend them to lower-resource settings (Dai and Le, 2015; Kiros et al., 2015; Bajgar et al., 2016).”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, background): “, 2015) and Book Test (BT) (Bajgar et al., 2016) are constructed in a similar manner.”
NewsQA: A Machine Comprehension Dataset (2016, background): “…for training data-intensive deep learning models, while those that are sufﬁciently large for deep learning (Hermann et al., 2015; Hill et al., 2016; Bajgar et al., 2016) are generated synthetically, yielding questions that are not posed in natural language and that may not test comprehension…”
The NarrativeQA Reading Comprehension Challenge (2017, methodology): “e-form, based on highlights entities Children’s Book Test (CBT) (Hill et al., 2016) 687K of 20 sentence passages from 108 children’s books Cloze-form, from the 21st sentence multiple choice BookTest (Bajgar et al., 2016) 14.2M, similar to CBT Cloze-form, similar to CBT multiple choice SQuAD (Rajpurk”
Gated-Attention Readers for Text Comprehension (2016, methodology): “For CBT-NE, GA Reader with the qecomm feature outperforms all previous single and ensemble models except the AS Reader trained on the much larger BookTest Corpus (Bajgar et al., 2016).”
DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension (2019, background): “…we have seen increased interest in large-scale cloze/span-based reading comprehension dataset construction (Hermann et al., 2015; Hill et al., 2016; Onishi et al., 2016; Rajpurkar et al., 2016; Bajgar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Choi et al., 2018).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “While there are many existing multiple-choice QA datasets (Richardson et al., 2013; Hill et al., 2015; Lai et al., 2017; Bajgar et al., 2016; Huang et al., 2019), these datasets use much shorter context passages (<500 tokens) than our dataset.”
BioMRC: A Dataset for Biomedical Machine Reading Comprehension (2020, methodology): “…domain, several cloze-style open-domain MRC datasets have been created automatically (Hill et al., 2016; Hermann et al., 2015; Dunn et al., 2017; Bajgar et al., 2016), but have been criticized of containing questions that can be answered by simple heuristics like our basic baselines (Chen et…”
CliCR: a Dataset of Clinical Case Reports for Machine Reading Comprehension (2018, methodology): “…(Hewlett et al., 2016; Joshi et al., 2017; Rajpurkar et al., 2016), web search queries (Nguyen et al., 2016), news articles (Hermann et al., 2015; Onishi et al., 2016; Trischler et al., 2017), books (Bajgar et al., 2016; Hill et al., 2016; Paperno et al., 2016) and English exams (Lai et al., 2017).”
Probing Prior Knowledge Needed in Challenging Chinese Machine Reading Comprehension (2019, background): “Much of the early work focuses on constructing large-scale extractive MRC datasets: Answers are spans from the reference document (Hermann et al., 2015; Hill et al., 2016; Bajgar et al., 2016; Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
Language Models are Unsupervised Multitask Learners (2019, result): “Bajgar et al. (2016) also previously improved results on the Children’s Book Test by creating a much larger training dataset out of Project Gutenberg to supplement the standard training dataset.”
From Particular to General: A Preliminary Case Study of Transfer Learning in Reading Comprehension (2016, background): “…Sum Reader (AS Reader) [10] can learn from training on one of the recently popular text-comprehension tasks – the Children’s Book Test (CBT) [9] – and its much larger extension – the BookTest (BT) [1] – thanks to which the model can give state-of-the-art performance on the CBT test data [1].”
LifeQA: A Real-life Dataset for Video Question Answering (2020, background): “With that goal in mind, several large-scale reading comprehension datasets were introduced (Rajpurkar et al., 2016; Richardson et al., 2013; Bajgar et al., 2016; Nguyen et al., 2016).”
Evidence Extraction for Machine Reading Comprehension with Deep Probabilistic Logic (2018, background): “Recently we have seen increased interest in extractive reading comprehension datasets (Hermann et al., 2015; Bajgar et al., 2016; Rajpurkar et al., 2016; Trischler et al., 2017).”
Simple and Effective Semi-Supervised Question Answering (2018, methodology): “Wiese et al. (2017) used SQuAD as a source for the target BioASQ dataset, and Kadlec et al. (2016) used Book Test (Bajgar et al., 2016) as source for the target SQuAD dataset.”