CoQA: A Conversational Question Answering Challenge

By Siva Reddy, Danqi Chen, Christopher D. Manning. 2018

1,430 citations recorded by the source corpus.

Humans gather information through conversations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essential to enable them to answer conversational questions. We introduce CoQA, a novel dataset for building Conversational Question Answering systems. Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets (e.g., coreference and pragmatic reasoning). We evaluate strong dialogue and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating that there is ample room for improvement. We present CoQA as a challenge to the community at https://stanfordnlp.github.io/coqa.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Language Models are Few-Shot Learners (2020, background): “serve GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset. GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “Generative question answering was mostly considered in previous work for datasets requiring to generate answers, such as NarrativeQA (Koˇcisk`y et al., 2018), CoQA (Reddy et al., 2019) or ELI5 (Fan et al., 2019).”
Technical report on Conversational Question Answering (2019, methodology): “Empirical results on the CoQA dataset show the effectiveness of our system, which outperforms the previous best model by a 2.6% absolute improvement in F1 score.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Yes/No questions make up a subset of the reading comprehension datasets CoQA (Reddy et al., 2018), QuAC (Choi et al., 2018), and Hot-PotQA (Yang et al., 2018), and are present in the ShARC (Saeidi et al., 2018) dataset.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “All of these datasets throw in additional complexities to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2019; Choi et al., 2018), requiring passage retrieval (Joshi et al.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
QuAC: Question Answering in Context (2018, methodology): “CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUA”
Improving Question Answering with External Knowledge (2019, background): “…numerous datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Lai et al., 2017; Mihaylov et al., 2018; Clark et al., 2018; Choi et al., 2018; Reddy et al., 2018; Sun et al., 2019) and models (Chen et al., 2016; Wang et al., 2018b; Radford et al., 2018; Devlin et al., 2018; Sun et al., 2018)…”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, neutral): “These tasks are TriviaQA, NQS, WebQS, SQuADv2, LAMBADA, DROP, QuAC and CoQA.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Conversational question-answering: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) 2.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “CoQA (Reddy et al., 2019) 8.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Affixes also vary by grammatical number: singular, dual (two), and plural (Ryding, 2005).”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Recent QA datasets have extended to a conversational form with a series of questions possibly referencing earlier ones in the conversation (Choi et al., 2018; Reddy, Chen, and Manning, 2019).”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Conversational benchmarks , like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.”
Language Models as Knowledge Bases? (2019, background): “They ﬁnd that GPTv2 achieves an F 1 of 55 for answering questions in CoQA (Reddy et al., 2018) and 4 .”