DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension

By Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, Karthik Sankaranarayanan. 2018

206 citations recorded by the source corpus.

We propose DuoRC, a novel dataset for Reading Comprehension (RC) that motivates several new challenges for neural approaches in language understanding beyond those offered by existing RC datasets. DuoRC contains 186,089 unique question-answer pairs created from a collection of 7680 pairs of movie plots where each pair in the collection reflects two versions of the same movie - one from Wikipedia and the other from IMDb - written by two different authors. We asked crowdsourced workers to create questions from one version of the plot and a different set of workers to extract or synthesize answers from the other version. This unique characteristic of DuoRC where questions and answers are created from different versions of a document narrating the same underlying story, ensures by design, that there is very little lexical overlap between the questions created from one version and the segments containing the answer in the other version. Further, since the two versions have different levels of plot detail, narration style, vocabulary, etc., answering questions from the second version requires deeper language understanding and incorporating external background knowledge. Additionally, the narrative style of passages arising from movie plots (as opposed to typical descriptive passages in existing datasets) exhibits the need to perform complex reasoning over events across multiple sentences. Indeed, we observe that state-of-the-art neural RC models which have achieved near human performance on the SQuAD dataset, even when coupled with traditional NLP techniques to address the challenges presented in DuoRC exhibit very poor performance (F1 score of 37.42% on DuoRC v/s 86% on SQuAD dataset). This opens up several interesting research avenues wherein DuoRC could complement other RC datasets to explore novel neural approaches for studying language understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “For this task, several different question answering datasets (Dua et al., 2019; Saha et al., 2018; Lai et al., 2017; Levy et al., 2017; Kembhavi et al., 2017; Rajpurkar et al., 2016) are converted into a singular, extractive format.”
What Makes Reading Comprehension Questions Easier? (2018, neutral): “0) (Rajpurkar et al., 2018), and DuoRC (Saha et al., 2018).”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “DuoRC (Saha et al., 2018) We use the Para-phraseRC split of the DuoRC dataset.”
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets (2019, methodology): “When training a model on DuoRC and HotpotQA, we used the ﬁrst answer span; i.e., the document spans that have no answer span were not used in training.”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “…WikiHop (Welbl et al., 2018); DuoRC Self (DuoRC-s) and DuoRC Paraphrase (DuoRCp) datasets (Saha et al., 2018) where the questions are taken from either the same version or a different version of the document from which the questions…”
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering (2019, neutral): “90 DuoRC (Saha et al., 2018) 53.”
On Making Reading Comprehension More Comprehensive (2019, methodology): “…include N ARRA - TIVE QA (Koˇcisk`y et al., 2018), where question authors were shown a summary of a movie script that will be used for answering questions, and D UO RC (Saha et al., 2018), where questions are authored given a passage that is comparable to the one that will later be employed.”
SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval (2020, background): “, 2019) Wikipedia DuoRC (Saha et al., 2018) Wikipedia+IMDB RACE (Lai et al.”
Comprehensive Multi-Dataset Evaluation of Reading Comprehension (2019, background): “DuoRC (Saha et al., 2018) tests if the model can generalize to answering semantically similar but syntactically different paraphrased questions.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, background): “…used to train the question generation model and teacher, and 6 outof-domain datasets—BioASQ (Tsatsaronis et al., 2015), DROP (Dua et al., 2019), DuoRC (Saha et al., 2018), RACE (Lai et al., 2017), RelationExtraction (Levy et al., 2017), and TextbookQA (Kembhavi et al., 2017)—for which no training…”
Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering (2021, background): “0 Multiple choices A piece of video caption DuoRC [137] Others 186.”
Domain-agnostic Question-Answering with Adversarial Training (2019, methodology): “…model for MRQA Shared Task with 6 different out-of-domain datasets, which are BioASQ (BA) (Tsatsaronis et al., 2012), DROP (DP) (Dua et al., 2019), DuoRC (DR) (Saha et al., 2018), RACE (RA) (Lai et al., 2017), RelationExtraction (RE) (Levy et al., 2017), and TextbookQA (TQ) (Kembhavi et al., 2017).”