MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text

By Matthew Richardson, C. Burges, Erin Renshaw. 2013

771 citations recorded by the source corpus.

We present MCTest, a freely available set of stories and associated questions intended for research on the machine comprehension of text. Previous work on machine comprehension (e.g., semantic modeling) has made great strides, but primarily focuses either on limited-domain datasets, or on solving a more restricted goal (e.g., open-domain relation extraction). In contrast, MCTest requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension. Reading comprehension can test advanced abilities such as causal reasoning and understanding the world, yet, by being multiple-choice, still provide a clear metric. By being fictional, the answer typically can be found only in the story itself. The stories and questions are also carefully limited to those a young child would understand, reducing the world knowledge that is required for the task. We present the scalable crowd-sourcing methods that allow us to cheaply construct a dataset of 500 stories and 2000 questions. By screening workers (with grammar tests) and stories (with grading), we have ensured that the data is the same quality as another set that we manually edited, but at one tenth the editing cost. By being open-domain, yet carefully restricted, we hope MCTest will serve to encourage research and provide a clear metric for advancement on the machine comprehension of text. 1 Reading Comprehension A major goal for NLP is for machines to be able to understand text as well as people. Several research disciplines are focused on this problem: for example, information extraction, relation extraction, semantic role labeling, and recognizing textual entailment. Yet these techniques are necessarily evaluated individually, rather than by how much they advance us towards the end goal. On the other hand, the goal of semantic parsing is the machine comprehension of text (MCT), yet its evaluation requires adherence to a specific knowledge representation, and it is currently unclear what the best representation is, for open-domain text. We believe that it is useful to directly tackle the top-level task of MCT. For this, we need a way to measure progress. One common method for evaluating someone’s understanding of text is by giving them a multiple-choice reading comprehension test. This has the advantage that it is objectively gradable (vs. essays) yet may test a range of abilities such as causal or counterfactual reasoning, inference among relations, or just basic understanding of the world in which the passage is set. Therefore, we propose a multiple-choice reading comprehension task as a way to evaluate progress on MCT. We have built a reading comprehension dataset containing 500 fictional stories, with 4 multiple choice questions per story. It was built using methods which can easily scale to at least 5000 stories, since the stories were created, and the curation was done, using crowd sourcing almost entirely, at a total of $4.00 per story. We plan to periodically update the dataset to ensure that methods are not overfitting to the existing data. The dataset is open-domain, yet restricted to concepts and words that a 7 year old is expected to understand. This task is still beyond the capability of today’s computers and algorithms.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “Finally, we present baseline experiments on the TriviaQA dataset, including a linear classiﬁer inspired by work on CNN Dailymail and MCTest (Chen et al., 2016; Richardson et al., 2013) and a state-of-the-art neural network baseline (Seo et al., 2017).”
Bidirectional Attention Flow for Machine Comprehension (2016, background): “Early datasets such as MCTest (Richardson et al., 2013) were too small to train end to end neural models.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, methodology): “In addition to the basic sliding window approach, we also implemented the distance-based extension (Richardson et al., 2013).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “The last, which we call transfer learning , leverages additional training examples from other multiple-choice QA tasks provided by the MMLU authors, namely MCTest (Richardson, 2013), RACE (Lai et al., 2017), ARC (Clark et al., 2018) and OBQA (Mihaylov et al., 2018) Additional benchmarks.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “If the question includes a context paragraph, it is appended after the candidate answers: question \n (A) c1 (B) c2 . . . \n paragraph , as shown in the example from the MCTest dataset.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, methodology): “These datasets have taken various forms: multiple-choice (Richardson et al., 2013), cloze-style (Hermann et al., 2015; Onishi et al., 2016; Hill et al., 2016), and span prediction (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) However, analysis (Chen et al., 2016; Sug-awara et…”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, methodology): “Finally, some datasets, like MCTest (Richard-son et al., 2013) and RACE (Lai et al., 2017), pose multiple choice questions, which can have a “none of the above” option.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, methodology): “Fiction: [277] • Stories from the Gutenberg project • Children stories from MCTest (Richard- son et al., 2013) • Movie plots from CMU Movie Sum- mary corpus (Bamman et al., 2013) From each of the above-mentioned sources we extracted paragraphs that had enough content.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Earlier reading comprehension datasets, e.g., MCTest (Richardson, 2013), SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al., 2016), and CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly sta”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “Although efforts have been made with a similar motivation, including the MCTest dataset created by (Richardson et al., 2013) (containing 500 passages and 2000 questions) and several others (Peñas et al.”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “Of course, text understanding can be tested through other tasks, including entailment detection (Bowman et al., 2015), answering questions about a text (Richardson et al., 2013; Weston et al., 2015) and measuring inter-clause coherence (Yin and Sch¨utze, 2015).”
NewsQA: A Machine Comprehension Dataset (2016, background): “Like Richardson et al. (2013) we want to encourage reasoning in comprehension models.”
Improving Question Answering with External Knowledge (2019, background): “Recent years have seen numerous datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Lai et al., 2017; Mihaylov et al., 2018; Clark et al., 2018; Choi et al., 2018; Reddy et al., 2018; Sun et al., 2019) and models (Chen et al.”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, background): “, 2018), MCTest (Richardson et al., 2013), COIN Shared Task 1 (Ostermann et al.”
Measuring Massive Multitask Language Understanding (2020, background): “, 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017; Richardson et al., 2013).”
MAWPS: A Math Word Problem Repository (2016, methodology): “In contrast to Richardson et al. (2013), our work is focused on making a dataset for math word problems to evaluate robustness, scalability, and scope of algorithms in quantitative reasoning.”
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories (2016, background): “There are a few recent frameworks for evaluating language comprehension (Hermann et al., 2015; Weston et al., 2015), including the MCTest (Richardson et al., 2013) as a notable one.”