LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

By Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao, Jian-Feng Gao. 2023

1,910 citations recorded by the source corpus.

Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-language models still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-language model using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Improved Baselines with Visual Instruction Tuning (2023, background): “The LLaVA architecture is also lever-aged in different downstream tasks and domains, including region-level [8, 55] and pixel-level [25, 49] understanding, biomedical assistants [30], image generation [5], adversarial studies [6, 58].”
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model (2023, background): “Multimodal LLMs have been attracting increasing interest from various research communities, such as computer vision [20], [21], embodied AI [22], [23], and biomedicine [24], [25].”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, positive): “Similarly, PMC-VQA [85] and LLaVA-Med [84] include text and image modalities for medical visual question answering.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, positive): “19) LLaVA-Med: LLaVA-Med [193] is a cost-efficient approach for training a vision-language conversational assistant based LLaVA [237] that can answer open-ended research questions of biomedical images.”
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, background): “Some of these models utilize a wide array of datasets collected from medical and biomedical literature, fine-tuned with specialized or open-ended instruction data (Li et al., 2023a; Singhal et al., 2023b).”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “Finally, we emloy a certain amount of image-only and text-only data from LLaVA [38], LLaVA-OneVision [28], Magpie [94], and Tulu 3 [111] for mitigating the impact of catastrophic forgetting on the model’s capabilities.”
HealthBench: Evaluating Large Language Models Towards Improved Human Health (2025, neutral): “…models, there has been increased interest in their application in healthcare (Clusmann et al., 2023; Thirunavukarasu et al., 2023; Lee et al., 2023; Liu et al., 2023; Moor et al., 2023; Singhal et al., 2023; Omiye et al., 2024; Zakka et al., 2024; Nori et al., 2023; Cosentino et al., 2024; Shah et…”
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, neutral): “In addition, some efforts attempt to adapt MLLMs for domain-specific tasks, such as medical image understanding [15, 17, 50] and document analysis [20, 44].”
Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, background): “Existing medical VLMs, such as MedVINT (Zhang et al., 2023b), are typically trained on paired image-text data with a single image in the context, as opposed to more general streams of text that are interleaved with multiple images.”
GeoChat:Grounded Large Vision-Language Model for Remote Sensing (2023, methodology): “More recent studies have shown that these models can be adapted to other domains such as videos [22], biomedical [14, 29] and remote sensing [12].”
LVLM-EHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models (2023, positive): “This methodsimpliﬁesthealignmentofimageandtextfeatures withinthesamesemanticspace,anditseffectivenesshasbeen demonstratedbyLLaVA[7].”
A survey on multimodal large language models (2023, neutral): “InstructBLIP [23], Ying-VLM [36], LLaVA-Med [37], MiniGPT-4 [13], X-LLM [25], MultiInstruct [26] GPT-aided PMC-VQA [30], VideoChat [33], DetGPT [38], LAMM [39], VisionLLM [24], MultiModal-GPT [31], LLaVA [21], ChatBridge [29], LLaVA-Med [37], GPT4Tools [34], Video-ChatGPT [40], Macaw-LLM [41]…”
Towards Generalist Biomedical AI (2023, positive): “LLaVA-Med [47] is perhaps most similar to our effort.”
A generalist vision–language foundation model for diverse biomedical tasks (2023, positive): “Figure 4c presents the prediction-accuracy results for five models, demonstrating that all three versions of BiomedGPT outperformed BioGPT and LLaVA-Med.”
EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, background): “The extraordinary potential of MLLMs is also evident in fields such as LIDAR [34] and robotics [35].”