Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets

By Saku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko Aizawa. 2019

92 citations recorded by the source corpus.

Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of the baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2% and 78.5% of the original scores, respectively. These results suggest that most of the questions already answered correctly by the model do not necessarily require grammatical and complex reasoning. For precise benchmarking, MRC datasets will need to take extra care in their design to ensure that questions can correctly evaluate the intended skills.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “Nonetheless, a growing body of research shows that resulting datasets may not isolate the key linguistic phenomena (Jia and Liang, 2017; Chen et al., 2016; Sugawara et al., 2020).”
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension (2020, background): “…seeks to understand the types of knowledge, comprehension skills or types of reasoning required to answer questions based on text (Rajpurkar et al., 2016; Clark et al., 2018; Sugawara et al., 2019; Dua et al., 2019; Dasigi et al., 2019); we are however unaware of any commonly accepted formalism.”
Dynabench: Rethinking Benchmarking in NLP (2021, neutral): “…competencies (Geiger et al., 2018; Naik et al., 2018; Glockner et al., 2018; White et al., 2018; Warstadt et al., 2019; Gauthier et al., 2020; Hossain et al., 2020; Jeretic et al., 2020; Lewis et al., 2020; Saha et al., 2020; Schuster et al., 2020; Sugawara et al., 2020; Warstadt et al., 2020).”
Data and its (dis)contents: A survey of dataset development and use in machine learning research (2020, background): “Sugawara et al. [2020] show that models trained on instances from machine reading comprehension datasets with scrambled words can still get the ‘right’ answer.”
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets (2020, background): “Examining what kinds of behaviours are learnt by models has received attention in natural langauge understanding tasks, such as the GLUE benchmark (Wang et al., 2018), which includes a diagnostic test set probing for different reasoning types.”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “• An evaluation dataset should be sufﬁciently free of annotation artifacts (as in Si et al., 2019; Sugawara et al., 2020b; Niven and Kao, 2019) that a system cannot reach near-human levels of performance by any means other than demonstrating the required language-related behaviors.”
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2020, background): “…in other NLP tasks ML models have been shown to be insensitive to word order in several NLP tasks such as reading comprehension (Si et al., 2019; Sugawara et al., 2020), dialog (Sankar et al., 2019), natural language inference (Parikh et al., 2016; Sinha et al., 2020), and essay scoring (Parekh…”
What Question Answering can Learn from Trivia Nerds (2019, background): “Sugawara et al. (2020) find that ablations like shuffling word order (Feng et al., 2018), shuffling sentences, or only offering the most similar sentence do not impair systems.”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, methodology): “Using them, Sugawara et al. (2020) reach performance comparable to that of a model that is trained on full input on a variety of MRC datasets.”
Rissanen Data Analysis: Examining Dataset Characteristics via Description Length (2021, methodology): “Another group of words that Sugawara et al. (2020) hypothesized are useful are words that express causal relationships: as, because, cause, reason, since, therefore, and why.”