Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

By Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicoló Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu and 8 more. arXiv.org, 2023

542 citations recorded by the source corpus.

Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/bde9da9a39a065588d7f4573936731510d6f4f29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards conversational diagnostic artificial intelligence (2025, unknown): “The scenario packs from the United Kingdom used in the OSCE study are also available for download from https://www. thefederation.uk/sites/default/files/documents/Station%202%20 Scenario%20Pack%20%2816%29.pdf.”
Empowering Biomedical Discovery with AI Agents (2024, unknown): “Early results in clinical medicine question-answering suggest that assigning speciﬁc roles, such as clinicians, to GPT-4 [61] can achieve better performance in terms of accuracy on multiple-choice benchmarks compared to using domain-specialized LLMs like BioGPT [63], NYUTron [64], and Med-PaLM [65,…”
Benchmarking Retrieval-Augmented Generation for Medicine (2024, methodology): “5 and GPT-4 (Ope-nAI et al., 2023) are two popular commercial LLMs developed by OpenAI, which have already shown great capabilities in answering medical questions (Nori et al., 2023b; Liévin et al., 2022).”
Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering (2024, background): “Due to the unique nature of code generation tasks, common prompting techniques that have been optimized for natural language tasks [4, 13, 10], may not be as effective when applied to code generation.”
GPT-4o System Card (2024, background): “In recent years, large language models have shown significant promise in biomedical settings, both in academic evaluations [39, 40, 41, 42, 43] and real-world use-cases such as clinical documentation [44, 45], patient messaging [46, 47], clinical trial recruitment [48, 49], and clinical decision…”
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations (2025, unknown): “…2024; ¨Ust¨un et al., 2024; Team et al., 2023) and similar improvements in specialized domains such as medicine (Thirunavukarasu et al., 2023; Nori et al., 2023; Abd-Alrazaq et al., 2023), radiology (D’Antonoli et al., 2024; Kim et al., 2024), law (Sun, 2023; Siino et al., 2025), etc. (Lo,…”