What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
By Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, Peter Szolovits. 2020
1,893 citations recorded by the source corpus.
Open domain question answering (OpenQA) tasks have been recently attracting more and more attention from the natural language processing (NLP) community. In this work, we present the first free-form multiple-choice OpenQA dataset for solving medical problems, MedQA, collected from the professional medical board exams. It covers three languages: English, simplified Chinese, and traditional Chinese, and contains 12,723, 34,251, and 14,123 questions for the three languages, respectively. We implement both rule-based and popular neural methods by sequentially combining a document retriever and a machine comprehension model. Through experiments, we find that even the current best method can only achieve 36.7%, 42.0%, and 70.1% of test accuracy on the English, traditional Chinese, and simplified Chinese questions, respectively. We expect MedQA to present great challenges to existing OpenQA systems and hope that it can serve as a platform to promote much stronger OpenQA models from the NLP community in the future.
Subfields: Artificial Intelligence, Question Answering & Reading Comprehension, Medical & Health AI
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Yoshua Bengio
- Quoc V. Le
- Furu Wei
In Question Answering & Reading Comprehension
- Benjamin Van Durme
- Daniel Khashabi
- Li Dong
- Furu Wei
- Jason Wei
- Christopher D. Manning
- Percy Liang
- Jure Leskovec
In Medical & Health AI
- Christopher D. Manning
- Percy Liang
- Jure Leskovec
- Michihiro Yasunaga
- Antoine Bosselut
- Hongyu Ren
- Xikun Zhang
Citations in the corpus
- Large language models encode clinical knowledge (2022, methodology): “These include the MedQA [33], MedMCQA [64], PubMedQA [34], LiveQA [1], MedicationQA [2] and MMLU clinical topics [29] datasets.”
- Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, background): “Specifically, the benchmarks include the following: • MedQA [14] contains multiple choice questions in the style of the Medical Licensing Examination questions used to test medical specialist competency in the United States, Mainland China, and Taiwan.”
- LinkBERT: Pretraining Language Models with Document Links (2022, methodology): “We follow the fine-tuning method in Jin et al. (2021).”
- QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, methodology): “, 2018) in the commonsense domain (using the ConceptNet KG), and MedQA-USMLE (Jin et al., 2021) in the biomedical domain (using the UMLS and DrugBank KGs).”
- Towards conversational diagnostic artificial intelligence (2025, positive): “We used the MedQA (multiple-choice) dataset, consisting of US Medical Licensing Examination multiple-choice-style open-domain questions with four or five possible answers 48 .”
- Baichuan 2: Open Large-scale Language Models (2023, methodology): “In the medical field, we report scores from two medical benchmarks, MedQA (Jin et al., 2021) and MedMCQA (Pal et al., 2022), as well as average scores from medical-related disciplines in C-Eval (val), MMLU, and CMMLU (abbreviated as CMC ).”
- Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “As shown in Figure 1, M IRAGE contains five commonly used datasets for medical QA for the evaluation of RAG systems (Hendrycks et al., 2020; Jin et al., 2021; Pal et al., 2022; Jin et al., 2019; Tsatsaronis et al., 2015), including three medical examination QA datasets (MMLU-Med, MedQA-US, MedMCQA)…”
- Graph Retrieval-Augmented Generation: A Survey (2024, neutral): “[102],[142],[181],[168],[42],[155],[58],[60],[101],[152],[3],[136],[90],[108],[139],[179],[23],[35],[100],[4],[133],[46],[61],[17],[140],[62] Accuracy,Hits@1,EM, Recall,F1, BERTScore, GPT-4AverageRanking WebQ[5] [159],[142],[52],[168],[107],[62] CWQ[144] [102],[142],[52],[181],[155],[60],[101],[107]”
- MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data (2023, positive): “Training data from the MedQA benchmark, question answering datasets consisting of medical exam questions [8] 4.”
- Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, background): “A wide variety of biomedical QA datasets have been introduced over the past decade, including BioASQ [60, 61], MedMCQA [44], MedQA (USMLE) [43], PubMedQA [62], GeneTuring [63].”
- Toward expert-level medical question answering with large language models (2025, neutral): “…language models trained using domain-specific data (BioLinkBert 10 , DRAGON 11 , PubMedGPT 12 , PubMedBERT 13 , BioGPT 14 ), resulting in steady improvements in performance on benchmark datasets such as MedQA (United States Medical Licensing Examination (USMLE)) 15 , MedMCQA 16 and PubMedQA 17 .”
- Deep Bidirectional Language-Knowledge Graph Pretraining (2022, neutral): “Similarly, as seen in §3.4.1, D RAGON also attains large gains over GreaseLM on complex questions containing negation, conjunction and prepositional phrases (Table 2), and extrapolates to questions more complex than seen in training sets (Figure 2).”
- MEDITRON-70B: Scaling Medical Pretraining for Large Language Models (2023, methodology): “We assess the downstream performance of the trial models by evaluating the finetuned models on the training sets of PubMedQA, MedMCQA, and MedQA.”
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, methodology): “The number k of options is 4 except for PubMedQA (3).”
- A study of generative large language model for medical research and healthcare (2023, positive): “These datasets have been widely used to evaluate LLMs 18,47 – 49 .”