HealthBench: Evaluating Large Language Models Towards Improved Human Health
By Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel and 2 more. arXiv.org, 2025
344 citations recorded by the source corpus.
We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health.
View Paper: https://www.semanticscholar.org/paper/01ea7fbc2604679dd80bf6271a64dc19cc5f0390
Citations in the corpus
- A benchmark of expert-level academic questions to assess AI capabilities (2025, unknown): “We train all reviewers on the instructions and rubric below.”
- gpt-oss-120b&gpt-oss-20b Model Card (2025, unknown): “• Additional Capabilities : We additionally test important capabilities such as multilingual abilities and health knowledge with benchmarks such as MMMLU [25] and HealthBench [28].”
- General-purpose large language models outperform specialized clinical AI tools on medical benchmarks (2026, unknown): “HealthBench is an OpenAI-developed benchmark that relies on a small number of physicians for each rubric, and public documentation provides limited detail on its construction and evaluation 5 .”
- Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks (2025, unknown): “3 While major frontier models are well-benchmarked 1,2 , specialized tools marketed for clinical use remain under-studied.”
- Federation of Agents: A Semantics-Aware Communication Fabric for Large-Scale Agentic AI (2025, unknown): “We evaluate the FoA framework on OpenAI’s HealthBench Hard [50], a comprehensive benchmark for assessing language models in healthcare contexts.”