Benchmarking Retrieval-Augmented Generation for Medicine
By Guangzhi Xiong, Qiao Jin, Zhiyong Lu, Aidong Zhang. Annual Meeting of the Association for Computational Linguistics, 2024
563 citations recorded by the source corpus.
While large language models (LLMs) have achieved state-of-the-art performance on a wide range of medical question answering (QA) tasks, they still face challenges with hallucinations and outdated knowledge. Retrieval-augmented generation (RAG) is a promising solution and has been widely adopted. However, a RAG system can involve multiple flexible components, and there is a lack of best practices regarding the optimal RAG setting for various medical purposes. To systematically evaluate such systems, we propose the Medical Information Retrieval-Augmented Generation Evaluation (MIRAGE), a first-of-its-kind benchmark including 7,663 questions from five medical QA datasets. Using MIRAGE, we conducted large-scale experiments with over 1.8 trillion prompt tokens on 41 combinations of different corpora, retrievers, and backbone LLMs through the MedRAG toolkit introduced in this work. Overall, MedRAG improves the accuracy of six different LLMs by up to 18% over chain-of-thought prompting, elevating the performance of GPT-3.5 and Mixtral to GPT-4-level. Our results show that the combination of various medical corpora and retrievers achieves the best performance. In addition, we discovered a log-linear scaling property and the"lost-in-the-middle"effects in medical RAG. We believe our comprehensive evaluations can serve as practical guidelines for implementing RAG systems for medicine.
Subfields: Artificial Intelligence, Molecular Biology
View Paper: https://www.semanticscholar.org/paper/b798cf6af813638fab09a8af6ad0f3df6c241485
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Evaluation of Retrieval-Augmented Generation: A Survey (2024, background): “The latest RAG benchmarks [49,50] in the specific domains start to evaluate the robustness of various prompting engineering skills, including CoT, RaR, etc. Inference The final input string prepared in the prompting step is then passed on to the LLMs as input, which generates the output.”
- Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “MIRAGE [366] is a benchmark designed for assessing the application of RAG in the medical domain, focusing on the comparison and optimization of medical question-answering systems’ performance.”
- MM-BizRAG: Rethinking Multimodal Retrieval-Augmented Generation for General Purpose Enterprise Q&A (2026, unknown): “Yet these pipelines often lack standardization (Gao et al., 2025; Zhang et al., 2025) in document representation (Sarthi et al., 2024; Jin et al., 2025; Yu et al., 2024a) and ingestion design (Xiong et al., 2024; Asai et al., 2023).”
- Clinical Context Variables Collectively Rival Model Choice in Embedding-Based Retrieval: Multi-Corpus Benchmark Study (2026, unknown): “…Information Retrieval-Augmented Generation Evaluation (MIRAGE) benchmark evaluated 41 combinations of corpora, retrievers, and backbone LLMs on 7663 medical questions and answers (QA), finding that corpus selection significantly affected performance and that no single configuration dominated [9].”
- Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale (2026, unknown): “Both interventions have well-documented benefits on medical benchmarks [13, 12], but they are usually compared in different studies, with different backbones, different evaluation sets, and different retrieval pipelines.”
- RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (2025, unknown): “Despite rapid progress, GraphRAG systems for text are often evaluated under heterogeneous protocols, with varying graph construction methods, retrieval configurations, and even evaluation criteria.”
- Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generation (2024, methodology): “5 answers PubMedQA and Com-monsenseQA questions with an accuracy of 48 .”
- MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation (2026, unknown): “As RAG systems are increasingly deployed in high-stakes enterprise domains ranging from medical diagnosis (Xiong et al., 2024) to wind energy (Meyur et al., 2025), the imperative for rigorous, scenario-specific evaluation has intensified.”