MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
By Zeming Chen, Alejandro Hern'andez Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Kopf, Amirkeivan Mohtashami and 10 more. arXiv.org, 2023
399 citations recorded by the source corpus.
Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning capacities, the resulting models are either closed-source (e.g., PaLM, GPT-4) or limited in scale (<= 13B parameters), which restricts their abilities. In this work, we improve access to large-scale medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through our adaptation of Nvidia's Megatron-LM distributed trainer), and extends pretraining on a comprehensively curated medical corpus, including selected PubMed articles, abstracts, and internationally-recognized medical guidelines. Evaluations using four major medical benchmarks show significant performance gains over several state-of-the-art baselines before and after task-specific finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the best public baseline in its parameter class and 3% over the strongest baseline we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. We release our code for curating the medical pretraining corpus and the MEDITRON model weights to drive open-source development of more capable medical LLMs.
Subfields: Artificial Intelligence, Health Informatics
View Paper: https://www.semanticscholar.org/paper/ff5f0c5b6905a8c4b361a625b450e9ab417fa854
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “Pretraining and fine-tuning are fundamental processes in the development and optimization of LLMs, including multimodal ones [69].”
- Can large language models reason about medical questions? (2022, neutral): “39 ,”
- The TRIPOD-LLM reporting guideline for studies using large language models (2025, neutral): “…in several domains across administrative and healthcare delivery use cases, including generating drafts for patient communication, medical document summarization, question answering, information retrieval, medical diagnosis, treatment recommendations, patient education and medical education 1–5 .”
- Towards conversational diagnostic artificial intelligence (2025, neutral): “Our OSCE study involved 20 board-certified PCPs and 20 validated patient-actors, ten each from India and Canada, respectively, to partake in online text-based consultations (Extended Data Fig.”
- Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024, positive): “We note that Llama 2, Clinical Camel and Meditron have been shown to match and even exceed Chat-GPT performance on medical licensing exams and biomedical question answering tests 19,35 .”