A visual-language foundation model for computational pathology
By Ming Y. Lu, Bowen Chen, Drew F. K. Williamson, Richard J. Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, L. Le, Georg K. Gerber and 3 more. Nature Medicine, 2024
1,028 citations recorded by the source corpus.
The accelerated adoption of digital pathology and advances in deep learning have enabled the development of robust models for various pathology tasks across a diverse array of diseases and patient cohorts. However, model training is often difficult due to label scarcity in the medical domain, and a model’s usage is limited by the specific task and disease for which it is trained. Additionally, most models in histopathology leverage only image data, a stark contrast to how humans teach each other and reason about histopathologic entities. We introduce CONtrastive learning from Captions for Histopathology (CONCH), a visual-language foundation model developed using diverse sources of histopathology images, biomedical text and, notably, over 1.17 million image–caption pairs through task-agnostic pretraining. Evaluated on a suite of 14 diverse benchmarks, CONCH can be transferred to a wide range of downstream tasks involving histopathology images and/or text, achieving state-of-the-art performance on histology image classification, segmentation, captioning, and text-to-image and image-to-text retrieval. CONCH represents a substantial leap over concurrent visual-language pretrained systems for histopathology, with the potential to directly facilitate a wide array of machine learning-based workflows requiring minimal or no further supervised fine-tuning. Developed using diverse sources of histopathology images, biomedical text and over 1.17 million image–caption pairs, evaluated on a suite of 14 diverse benchmarks, a visual-language foundation model achieves state-of-the-art performance on a wide array of clinically relevant pathology tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Molecular Biology
View Paper: https://www.semanticscholar.org/paper/0746410e1250576cc1b9621ef602e4bceb1c07ca
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- A foundation model for clinical-grade computational pathology and rare cancers detection (2024, neutral): “…greater for applications with inadequate quantities of data to develop bespoke models, as is the case for the detection of uncommon or rare tumor types, as well as for less common diagnostic tasks such as the prediction of specific genomic alterations, clinical outcomes and therapeutic response.”
- The TRIPOD-LLM reporting guideline for studies using large language models (2025, neutral): “Of note, this guideline was developed with text-only LLMs in mind; however, advances in multimodal models incorporating LLMs, such as vision-language models 47 , are now rapidly emerging — illustrating the need for rapid, nimble approaches for reporting guidelines.”
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, positive): “In this section, we present a use case that demonstrates the practical application of M-LLMs in health care using the Contrastive Learning From Captions for Histopathology (CONCH) model [141].”