ExpertQA: Expert-Curated Questions and Attributed Answers

By Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, Dan Roth. 2023

113 citations recorded by the source corpus.

As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. In addition, we ask experts to improve upon responses from language models. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.

Subfields: Management Information Systems, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Question Answering & Reading Comprehension

Citations in the corpus

GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “By comparison, the ExpertQA dataset (Malaviya et al., 2023) also sources questions from experts, but the answers are determined by having experts revise language model outputs, and not checked against other experts’ answers or tested for difficulty for non-experts.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “Within four months since its initial release, FA CT S CORE has actively been used in subsequent work, evaluating factual precision of recently-proposed models (Ye et al., 2023; Sun et al., 2023; Malaviya et al., 2023; Dhuliawala et al., 2023).”