PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents
By Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie. International Conference on Medical Image Computing and Computer-Assisted Intervention, 2023
336 citations recorded by the source corpus.
Foundation models trained on large-scale dataset gain a recent surge in CV and NLP. In contrast, development in biomedical domain lags far behind due to data scarcity. To address this issue, we build and release PMC-OA, a biomedical dataset with 1.6M image-caption pairs collected from PubMedCentral's OpenAccess subset, which is 8 times larger than before. PMC-OA covers diverse modalities or diseases, with majority of the image-caption samples aligned at finer-grained level, i.e., subfigure and subcaption. While pretraining a CLIP-style model on PMC-OA, our model named PMC-CLIP achieves state-of-the-art results on various downstream tasks, including image-text retrieval on ROCO, MedMNIST image classification, Medical VQA, i.e. +8.1% R@10 on image-text retrieval, +3.9% accuracy on image classification.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/8f3138f7ee5127faab265793be8ae278bc49d9b1
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Michihiro Yasunaga
- Jure Leskovec
- Percy Liang
- Qian Huang
- Weidi Xie
- Pranav Rajpurkar
- Yanfeng Wang
- Ya Zhang
In Computer Vision and Pattern Recognition
- Jure Leskovec
- Michihiro Yasunaga
- Weidi Xie
- Qian Huang
- Pranav Rajpurkar
- Yanfeng Wang
- Ya Zhang
- Mohit Bansal
Citations in the corpus
- Med-Flamingo: a Multimodal Medical Few-shot Learner (2023, methodology): “As for the paired datasets, we used PMC-OA (Lin et al., 2023).”
- Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Interestingly, HELM pointed out that LMs such as T5 [95] and Anthropic-LMv4-s3 [5] had not been directly compared in the initial work, while LLMs such as GPT-3 and YaLM were still different from their corresponding reports after multiple evaluations.”
- PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering (2023, methodology): “Method Language Backbone Vision Backbone Blanking Choice ACC Bleu-1 ACC Zero-shot PMC-CLIP [20] PMC-CLIP [20] PMC-CLIP [20] – – 24.”
- A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, neutral): “Besides, PMC-OA dataset [244] was also employed, which consists of 1.”