DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension

By Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, Claire Cardie. 2019

319 citations recorded by the source corpus.

We present DREAM, the first dialogue-based multiple-choice reading comprehension data set. Collected from English as a Foreign Language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our data set contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension data sets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule-based approach. We next investigate the effects of incorporating dialogue structure and different kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehension models. Experimental results on the DREAM data set show the effectiveness of dialogue structure and general world knowledge. DREAM is available at https://dataset.org/dream/.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Improving Question Answering with External Knowledge (2019, methodology): “, 2019) since questions from these datasets are generally clean, error-free, and challenging (Sun et al., 2019).”
Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, methodology): “No model achieves the estimated human ceiling for either RACE (Lai et al., 2017) or DREAM (Sun et al., 2019).”
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (2022, neutral): “…et al., 2018), qasc (Khot et al., 2020), commonsense_qa (Talmor et al., 2019), ai2_arc (Clark et al., 2018), codah (Chen et al., 2019), superglue-copa (Gordon et al., 2012), dream (Sun et al., 2019), quartz-with_knowledge (Tafjord et al., 2019b), quartz-no_knowledge (Tafjord et al., 2019b).”
CoMPS: Continual Meta Policy Search (2021, methodology): “…et al., 2016; Mishra et al., 2017; Stadie et al., 2018; Rakelly et al., 2019; Zintgraf et al., 2020; Sæmundsson et al., 2018; Fakoor et al., 2019; Sun et al., 2019; Dorfman & Tamar, 2020), applying gradient descent from initial parameters (Finn et al., 2017; Gupta et al., 2018b; Rothfuss et…”
Reason first, then respond: Modular Generation for Knowledge-infused Dialogue (2021, background): “The DREAM dataset (Sun et al., 2019) consists of multiple-choice questions about a conversation.”
A Survey of Deep Learning for Mathematical Reasoning (2022, unknown): “mathematical reasoning grounded on conversations (Sun et al., 2019; Zhang et al., 2021; Chen et al., 2022c), as well as reports (Chen et al.”
Can You Unpack That? Learning to Rewrite Questions-in-Context (2019, background): “…avenue for future work is to incorporate deeper context, either from other modalities (Das et al., 2017) or from other dialog comprehension tasks (Sun et al., 2019 Training question rewriting using reinforcement learning with the task accuracy as reward signal is explored in retrieval-based QA…”
MuTual: A Dataset for Multi-Turn Dialogue Reasoning (2020, background): “Another related reading comprehension dataset is DREAM (Sun et al., 2019), which is designed specifically for challenging dialogue-based reading”
Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian (2019, methodology): “…extractive, e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g., multiple-choice questions (Richardson et al., 2013; Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019a).”
CIDER: Commonsense Inference for Dialogue Explanation and Reasoning (2021, background): “DREAM (Sun et al., 2019) is a dialogue-based multiple-choice reading-comprehension dataset collected from exams of English as a foreign language.”
ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning (2020, methodology): “This similar phenomenon can also be observed on DREAM dataset (Sun et al., 2019) by Jin et al.”
Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021, background): “DREAM (Sun et al., 2019) is a dialogue-based MCRC dataset, where the context is a multi-turn, multi-party dialogue.”
Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering (2019, methodology): “We evalutate our models on the DREAM (Sun et al., 2019) and CommonsenseQA (Talmor et al., 2019) datasets.”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, background): “…when considering information from multiple sentences and/or documents (Clark et al., 2018; Khashabi et al., 2018a; Yang et al., 2018; Welbl et al., 2018; Mihaylov et al., 2018; Bauer et al., 2018; Dunn et al., 2017; Dhingra et al., 2017; Lai et al., 2017; Rajpurkar et al., 2018; Sun et al., 2019).”
Improving Machine Reading Comprehension with Single-choice Decision and Transfer Learning (2020, background): “Generally, the matching network can be various (Ran et al., 2019; Zhang et al., 2020; Zhu et al., 2020).”
Evidence Sentence Extraction for Machine Reading Comprehension (2019, methodology): “, 2013), DREAM (Sun et al., 2019) and RACE (Lai et al.”