Comprehensive Multi-Dataset Evaluation of Reading Comprehension

By Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner. 2019

9 citations recorded by the source corpus.

Reading comprehension is one of the crucial tasks for furthering research in natural language understanding. A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context. Given the availability of many such datasets, comprehensive and reliable evaluation is tedious and time-consuming for researchers working on this problem. We present an evaluation server, ORB, that reports performance on seven diverse reading comprehension datasets, encouraging and facilitating testing a single model’s capability in understanding a wide variety of reading phenomena. The evaluation server places no restrictions on how models are trained, so it is a suitable test bed for exploring training paradigms and representation learning for general reading facility. As more suitable datasets are released, they will be added to the evaluation server. We also collect and include synthetic augmentations for these datasets, testing how well models can handle out-of-domain questions.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “In ORB, Dua et al. (2019a) propose a multi-dataset evaluation benchmark spanning extractive and abstractive formats.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, methodology): “We may also ask to evaluate a single model on multiple test sets, following the design by Dua et al. (2019).”
Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models (2020, background): “Those focus on prediction consistency (Ribeiro et al., 2019), acquired knowledge (Richardson and Sabharwal, 2019), unanswer-ability (Nakanishi et al., 2018) or multi-dataset evaluation (Dua et al., 2019a) rather than performing an analysis of acquired linguistic or reasoning capabilities.”
Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks (2020, methodology): “ORB [Dua et al., 2019a] has an evaluation server which reports performance on seven diverse reading comprehension datasets, encouraging development of a single model for a wide variety of reading phenomena.”