Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing

By Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel Coelho de Castro, Anton Schwaighofer, Stephanie L. Hyland, Maria Teodora Wetscherek, Tristan Naumann, A. Nori, Javier Alvarez-Valle and 2 more. European Conference on Computer Vision, 2022

489 citations recorded by the source corpus.

Multi-modal data abounds in biomedicine, such as radiology images and reports. Interpreting this data at scale is essential for improving clinical care and accelerating clinical research. Biomedical text with its complex semantics poses additional challenges in vision--language modelling compared to the general domain, and previous work has used insufficiently adapted models that lack domain-specific language understanding. In this paper, we show that principled textual semantic modelling can substantially improve contrastive learning in self-supervised vision--language processing. We release a language model that achieves state-of-the-art results in radiology natural language inference through its improved vocabulary and novel language pretraining objective leveraging semantics and discourse characteristics in radiology reports. Further, we propose a self-supervised joint vision--language approach with a focus on better text modelling. It establishes new state of the art results on a wide range of publicly available benchmarks, in part by leveraging our new domain-specific language model. We release a new dataset with locally-aligned phrase grounding annotations by radiologists to facilitate the study of complex semantic modelling in biomedical vision--language processing. A broad evaluation, including on this new dataset, shows that our contrastive learning approach, aided by textual-semantic modelling, outperforms prior methods in segmentation tasks, despite only using a global-alignment objective.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a1aeb5442e31276b197696f49b3243112a4049ce

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Chunyuan Li
3. Jianwei Yang
4. Haotian Liu
5. Hoifung Poon
6. Zhe Gan
7. Sheng Zhang
8. Percy Liang

In Computer Vision and Pattern Recognition

1. Chunyuan Li
2. Jianfeng Gao
3. Jianwei Yang
4. Haotian Liu
5. Luc Van Gool
6. Xiaogang Wang
7. Dacheng Tao
8. Trevor Darrell

Citations in the corpus

• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs (2023, background): “BioViL [18] improves contrastive learning in self-supervised vision-language processing with radiology-specific semantic modeling tailored to CXR images and reports.”
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day (2023, background): “Domain-specific pretraining has been shown to be effective for biomedical natural language processing (NLP) applications [18, 15, 11, 31] and biomedical vision-language (VL) tasks [16, 7, 41, 51, 9].”
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “Despite this, the existing VLP work to date considers alignment between only single images and reports [9,32,46,81], going so far as to remove temporal content from reports in training data to prevent ‘hallucinations’ in downstream report generation [54].”