Making Neural QA as Simple as Possible but not Simpler

By Dirk Weissenborn, Georg Wiese, Laura Seiffe. 2017

220 citations recorded by the source corpus.

Recent development of large-scale question answering (QA) datasets triggered a substantial amount of research into end-to-end neural architectures for QA. Increasingly complex systems have been conceived without comparison to simpler neural baseline systems that would justify their complexity. In this work, we propose a simple heuristic that guides the development of neural baseline systems for the extractive QA task. We find that there are two ingredients necessary for building a high-performing neural QA system: first, the awareness of question words while processing the context and second, a composition function that goes beyond simple bag-of-words modeling, such as recurrent neural networks. Our results show that FastQA, a system that meets these two requirements, can achieve very competitive performance compared with existing models. We argue that this surprising finding puts results of previous systems and the complexity of recent QA datasets into perspective.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional Published 12 LeaderBoard 13 Single Model EM / F1 EM / F1 LR Baseline (Rajpurkar et al., 2016) 40.4 / 51.0 40.4 / 51.0 Dynamic C”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “…47.06 49.54 55.80 MEMEN* (Pan et al., 2017) 43.16 46.90 49.28 55.83 Mnemonic Reader (Hu et al., 2017) 46.94 52.85 54.45 59.46 Reading Twice for NLU (Weissenborn et al., 2017a) approaches as the model is used to independently process an increasing number of these paragraphs and the model’s most…”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “…et al., 2016; Trischler et al., 2017; Joshi et al., 2017) However, analysis (Chen et al., 2016; Sug-awara et al., 2017) of these datasets has shown that many of the questions can be solved with context token matching (Chen et al., 2017a; Weissenborn et al., 2017) or relatively simple paraphrasing.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017), question answering (Seo et al., 2017; Xiong et al., 2018; Yu et al., 2016; Weissenborn et al., 2017), and curriculum learning (Bengio et al.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Otherwise, type-matching heuristics could distinguish answerable and unan-swerable questions (Weissenborn et al., 2017).”
ELI5: Long Form Question Answering (2019, neutral): “In contrast to previous datasets where the human written answer could be found with lexical overlap meth-ods (Weissenborn et al., 2017), ELI5 poses a signiﬁcant challenge in siphoning out important information, as no single sentence or phrase contains the full answer.”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “…is known to result in machine-exploitable annotator artefacts (Jia and Liang, 2017; Schwartz et al., 2017; Gururangan et al., 2018; Geva et al., 2019), leading to poor out-of-distribution generalisation (Chen et al., 2016; Weissenborn et al., 2017; Yogatama et al., 2019; McCoy et al., 2019).”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, background): “Weissenborn et al. (2017) argue that many SQuAD questions can be answered with heuristics based on type and keyword-matching.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, methodology): “For both datasets we then establish several strong baselines and evaluate the performance of two previously proposed competitive RC models (Seo et al., 2017a; Weissenborn et al., 2017).”
Stochastic Answer Networks for Machine Reading Comprehension (2017, background): “It is simple yet efﬁcient and can be trained using the classical back-propagation algorithm, thus it is adopted by most of systems (Chen et al., 2016; Seo et al., 2016; Wang et al., 2017; Liu et al., 2017; Chen et al., 2017; Weissenborn et al., 2017; Hu et al., 2017).”
The price of debiasing automatic metrics in natural language evalaution (2018, methodology): “We collected annotations on the systems generated by the fastqa and fastqa ext from Weissenborn et al. (2017) and the snet and snet.ens (emble) models from Tan et al. (2018), along with reference answers.”
What Makes Reading Comprehension Questions Easier? (2018, methodology): “Although similar heuristics are proposed by Weissenborn et al. (2017), ours are utilized for question ﬁltering, rather than system development; Using these simple heuristics, we split each dataset into easy and hard subsets for further investigation on the baseline performance.”
Adversarial Text Generation Without Reinforcement Learning (2018, background): “…several years, deep learning models have provided large performance gains on many tasks requiring language generation, from machine translation (Johnson et al., 2016) to dialogue agents (Serban et al., 2016) to summarization (Rush et al., 2015) and question answering (Weissenborn et al., 2017).”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, methodology): “, 2018) and FastQA (Weissenborn et al., 2017), which are the model leveraging sentence selection for question answering, and the published state-of-the-art models on SQuAD and NewsQA, respectively.”
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension (2017, background): “Some (Chen et al., 2017a; Weissenborn et al., 2017) attend the word-level embedding from the question to context, while some (Wang et al., 2017) attend the high-level representation in the question to augment the context.”
DCN+: Mixed Objective and Deep Residual Coattention for Question Answering (2017, background): “(Shen et al., 2017), Document Reader (Chen et al., 2017), FastQA (Weissenborn et al., 2017), DCN (Xiong et al., 2017).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, background): “Prior work has shown that a lot of questions in existing datasets such as SQuAD can be answered by exploiting lexical overlap of the question with the article (Weissenborn et al., 2017).”