What Makes Reading Comprehension Questions Easier?

By Saku Sugawara, Kentaro Inui, Satoshi Sekine, Akiko Aizawa. 2018

127 citations recorded by the source corpus.

A challenge in creating a dataset for machine reading comprehension (MRC) is to collect questions that require a sophisticated understanding of language to answer beyond using superficial cues. In this work, we investigate what makes questions easier across recent 12 MRC datasets with three question styles (answer extraction, description, and multiple choice). We propose to employ simple heuristics to split each dataset into easy and hard subsets and examine the performance of two baseline models for each of the subsets. We then manually annotate questions sampled from each subset with both validity and requisite reasoning skills to investigate which skills explain the difference between easy and hard questions. From this study, we observed that (i) the baseline performances for the hard subsets remarkably degrade compared to those of entire datasets, (ii) hard questions require knowledge inference and multiple-sentence reasoning in comparison with easy questions, and (iii) multiple-choice questions tend to require a broader range of reasoning skills than answer extraction and description questions. These results suggest that one might overestimate recent advances in MRC.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “Some works (Chen et al., 2016; Sugawara et al., 2018; Kaushik and Lipton, 2018) classify the reasoning skills required to answer the questions into the following types: (1) Exact matching/Paraphrasing; (2) Summary; (3) Logic reasoning; (4) Utilizing external knowledge; (5) Numerical reasoning.”
Improving Question Answering with External Knowledge (2019, background): “We refer readers to Sugawara et al. (2018) for detailed definitions of each category.”
Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “Other work has offered insight by splitting the data into “easy” and “hard” sets with respect to some attribute and studying changes in model performance, but these methods do not offer a point-wise estimate of difficulty (Sugawara et al., 2018; Rondeau & Hazen, 2018; Sen & Saffari, 2020).”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, background): “Wang et al. (2018a) concurrently introduced a neural model that extracts evidence specifically for the correct answer, as an intermediate step in a QA pipeline.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, methodology): “However, single sentence context questions are a known issue in SQuAD annotation in general (Sugawara et al., 2018) suggesting our method would not result in less challenging questions, supported by scores on MLQA-en being similar to SQuAD (section 5.3).”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “However, mounting evidence suggests that models can achieve strong generalisation performance merely by relying on superﬁcial cues – such as lexical overlap, term frequencies, or entity type matching (Chen et al., 2016; Weissenborn et al., 2017; Sugawara et al., 2018).”
Factual Probing Is [MASK]: Learning vs. Learning to Recall (2021, background): “McCoy et al. (2019) discover that models like BERT are often “right for the wrong reason”, exploiting shallow heuristics rather than underlying linguistic structure, and similar effects have been discovered in many other tasks (Sugawara et al., 2018; Wallace et al., 2019).”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “Speciﬁcally, Sugawara et al. (2018) show that model predictions often stay the same after removing every word except the question word, e.g., “when was the battle?””
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2020, background): “Whilst there have been several works examining other kinds of QA datasets (Manjunatha et al., 2018; Kaushik and Lipton, 2018; Sugawara et al., 2018, 2020), we know comparatively little about how the questions and answers are distributed in these ODQA benchmarks, making it hard to understand and…”
Compositional Questions Do Not Necessitate Multi-hop Reasoning (2019, methodology): “To further investigate entity type matching, we reduce the question to the ﬁrst ﬁve tokens starting from the wh-word, following Sugawara et al. (2018).”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, background): “, by simply attending to context tokens that are similar to those of the questions (Sugawara et al. 2018)).”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “…al., 2018; Poliak et al., 2018; McCoy et al., 2019), question-passage overlap and entity cues in reading comprehension datasets (Chen et al., 2016; Sugawara et al., 2018; Jia & Liang, 2017; Lewis et al., 2021), gender cues in coreference resolution datasets (Rudinger et al., 2018), popularity in…”
Unsupervised Question Answering by Cloze Translation (2019, background): “Wang et al. (2018) consider semi-supervised cloze QA, Chen et al. (2018) use semi-supervision to improve semantic parsing on WebQuestions (Berant et al., 2013), and Lei et al. (2016) leverage semi-supervision for question similarity modelling.”
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020, background): “This poses new challenges to standard models, which are known to exploit statistical patterns or annotation artifacts in these datasets (Sugawara et al., 2018; Min et al., 2019a).”
Competency Problems: On Finding and Removing Artifacts in Language Data (2021, background): “Attempts by the natural language processing community to get machines to understand language or read text are often stymied in part by issues in our datasets (Chen et al., 2016; Sugawara et al., 2018).”
What Do Models Learn from Question Answering Datasets? (2020, background): “In previous work, Agrawal et al. (2016) found that a Visual Question Answering (VQA) model could get good performance with just half the original question, while Sugawara et al. (2018) saw drops in BiDAF model performance on QA datasets with increasingly incomplete questions.”
Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction (2019, background): “However, Sugawara et al. (2018) indicated that the most questions in RC require reasoning from just one sentence including the answer.”