Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

By Jason Aaron Edward Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov. 2015

1,265 citations recorded by the source corpus.

One long-term goal of machine learning research is to produce methods that are applicable to reasoning and natural language, in particular building an intelligent dialogue agent. To measure progress towards that goal, we argue for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. We believe many existing learning systems can currently not solve them, and hence our aim is to classify these tasks into skill sets, so that researchers can identify (and then rectify) the failings of their systems. We also extend and improve the recently introduced Memory Networks model, and show it is able to solve some, but not all, of the tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Structured Attention Networks (2017, methodology): “We use the bAbI dataset (Weston et al., 2015), where the input is a set of sentences/facts paired with a question, and the answer is a single token.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Yes/No QA has been used in other contexts, such as the templated bAbI stories (Weston et al., 2015) or some Visual QA datasets (Antol et al., 2015; Wu et al., 2017).”
CoQA: A Conversational Question Answering Challenge (2018, background): “, 2015), using a hand-written grammar to create artificial questions (Weston et al., 2016; Welbl et al., 2018), paraphrasing artificial questions to natural questions (Saha et al.”
Universal Transformers (2018, background): “The bAbi question answering dataset [32] consists of 20 different tasks, where the goal is to answer a question given a number of English sentences that encode potentially multiple supporting facts.”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “Weston et al. (2015) present a relatively small dataset of 10 reasoning categories, and propose to build a system that uses a world model and a linguistic model.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “To push towards more complex QA tasks, one approach has been to generate synthetic datasets, the most notable example being the bAbI dataset (Weston et al., 2015).”
The LAMBADA dataset: Word prediction requiring a broad discourse context (2016, background): “Of course, text understanding can be tested through other tasks, including entailment detection (Bowman et al., 2015), answering questions about a text (Richardson et al., 2013; Weston et al., 2015) and measuring inter-clause coherence (Yin and Sch¨utze, 2015).”
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories (2016, background): “There are a few recent frameworks for evaluating language comprehension (Hermann et al., 2015; Weston et al., 2015), including the MCTest (Richardson et al., 2013) as a notable one.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “To do this one direction they have studied is artificial tasks involving controlled reasoning on toy problems in order to develop more sophisticated architectures (Weston et al., 2015).”
Large Memory Layers with Product Keys (2019, background): “For instance, memory-based neural layers [45, 41] are an efﬁcient way to represent variable length inputs for complex problems such as question answering [46].”
A Knowledge-Grounded Neural Conversation Model (2017, methodology): “Memory network models are widely used in Question Answering to make inferences based on the facts saved in the memory (Weston et al., 2015).”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “, 2013) is a smaller-scale datasets focusing on common sense reasoning; bAbi (Weston et al., 2015) is a synthetic dataset that captures various aspects of reasoning; and SQuAD (Rajpurkar et al.”
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016, background): “domain question answering is one of those domains existing systems still struggle to solve [31].”
The NarrativeQA Reading Comprehension Challenge (2017, background): “First, several datasets are small (MCTest) or not overly naturalistic (bAbI; Weston et al. (2015)).”
Key-Value Memory Networks for Directly Reading Documents (2016, background): “ulated stories (Weston et al., 2016) and for utilizing KBs to answer questions (Bordes et al.”