QuAC: Question Answering in Context

By Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, Luke S. Zettlemoyer. 2018

904 citations recorded by the source corpus.

We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “For reading comprehension, we report the 0-shot average on SQuAD (Rajpurkar et al., 2018), QuAC (Choi et al., 2018), and BoolQ (Clark et al., 2019).”
Mistral 7B (2023, neutral): “…[9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval [5] (0-shot) and MBPP [2] (3-shot) • Popular aggregated…”
Mixtral of Experts (2024, neutral): “…[8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval [4] (0-shot) and MBPP [1] (3-shot) • Popular aggregated…”
Technical report on Conversational Question Answering (2019, methodology): “Recently, several CQA datasets were proposed, such as CoQA(Reddy et al., 2018), QuAC (Choi et al., 2018) and QBLink (Elgohary et al., 2018).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “It was challenging to train annotators to do well on Quora Insincere Questions 10 , Empa-thetic Reactions (Buechel et al., 2018), and a recast version of Ultra-Fine Entity Typing (Choi et al., 2018b, see Appendix C.3 for details), leading to low human performance.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, background): “Yes/No questions do appear as a subset of some existing datasets (Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “All of these datasets throw in additional complexities to the reading comprehension challenge, around tracking conversational state (Reddy et al., 2019; Choi et al., 2018), requiring passage retrieval (Joshi et al.”
Learning and Evaluating General Linguistic Intelligence (2019, methodology): “We use two additional reading comprehension datasets: TriviaQA (Joshi et al., 2017) and QuAC 4 (Choi et al., 2018) Semantic Role Labeling.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
Improving Question Answering with External Knowledge (2019, background): “…years have seen numerous datasets (Richardson et al., 2013; Rajpurkar et al., 2016; Lai et al., 2017; Mihaylov et al., 2018; Clark et al., 2018; Choi et al., 2018; Reddy et al., 2018; Sun et al., 2019) and models (Chen et al., 2016; Wang et al., 2018b; Radford et al., 2018; Devlin et al.,…”
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (2021, neutral): “These tasks are TriviaQA, NQS, WebQS, SQuADv2, LAMBADA, DROP, QuAC and CoQA.”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Conversational question-answering: QuAC (Choi et al., 2018) and CoQA (Reddy et al., 2019) 2.”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Large scale datasets such as (Rajpurkar et al., 2016; Choi et al., 2018; Trischler et al., 2016) have driven fast progress.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of information-seeking questions such as WikiQA (Yang et al., 2015), MS MARCO (Nguyen et al., 2016), QuAC Pronounced tie dye Q. A. —like the colorful t-shirt.”
Datasheets for datasets (2018, neutral): “Academic researchers have adopted our proposal and released datasets with accompanying datasheets [e.g., 7, 10, 23, 26].”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Recent QA datasets have extended to a conversational form with a series of questions possibly referencing earlier ones in the conversation (Choi et al., 2018; Reddy, Chen, and Manning, 2019).”