Self-Alignment Pretraining for Biomedical Entity Representations

By Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, Nigel Collier. North American Chapter of the Association for Computational Linguistics, 2020

466 citations recorded by the source corpus.

Despite the widespread success of self-supervised learning via masked language models (MLM), accurately capturing fine-grained semantic relationships in the biomedical domain remains a challenge. This is of paramount importance for entity-level tasks such as entity linking where the ability to model entity relations (especially synonymy) is pivotal. To address this challenge, we propose SapBERT, a pretraining scheme that self-aligns the representation space of biomedical entities. We design a scalable metric learning framework that can leverage UMLS, a massive collection of biomedical ontologies with 4M+ concepts. In contrast with previous pipeline-based hybrid systems, SapBERT offers an elegant one-model-for-all solution to the problem of medical entity linking (MEL), achieving a new state-of-the-art (SOTA) on six MEL benchmarking datasets. In the scientific domain, we achieve SOTA even without task-specific supervision. With substantial improvement over various domain-specific pretrained MLMs such as BioBERT, SciBERTand and PubMedBERT, our pretraining scheme proves to be both effective and robust.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/bb3d3d4be92d741d2dc30e6cd02468f74be0d716

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, methodology): “…USMLE BioALBERT [168] FT PubMed, PMC, MIMIC-III 6 BioNLP Tasks BioBART [169] FT PubMed Biomedical EL, NER, QA, Dialogue, Summarization SAPBERT [170] FT UMLS MEL VPP [10] FT PubMed Biomedical NER KAD [171] FT MIMIC-CXR PadChest, ChestXray14, CheXpert and ChestX-Det10 A. PLMs for Healthcare…”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “One line of works is retrieval-augmented LMs (Guu et al. 2020; Lewis et al. 2020; Borgeaud et al. 2021), which retrieve relevant text from a corpus and integrate it into LMs as additional knowledge.”
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, methodology): “For MedQA-USMLE, we use a state-of-theart biomedical LM, SapBERT (Liu et al., 2020a).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “Clinical Kb-ALBERT [201], SapBERT [202], Sap-XLMR [202], UmlsBERT [127], CoderBERT [203], CoderBERTAll [203] in specific domain like Biomedical.”
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021, unknown): “While fine-tuning on NLI (Sentence-BERT) or UMLS (SapBERT) data can take hours, Mirror-BERT with 10k positive pairs completes the conversion from MLMs to universal language encoders within a minute on two NVIDIA RTX 2080Ti GPUs.”
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, methodology): “…with RoBERTa-Large (Liu et al., 2019) for our experiments on CommonsenseQA, AristoRoBERTa (Clark et al., 2019) for our experiments on Open-bookQA, and SapBERT (Liu et al., 2021) for our experiments on MedQA-USMLE, demonstrating G REASE LM’s generality with respect to language model initializations.”
RELATE: Relation Extraction in Biomedical Abstracts with LLMs and Ontology Constraints (2025, unknown): “To address limitations of general-purpose embedding models (e.g., nomic-text-embed or bge-m3 ) in biomedical domains, we introduce an (optional) hybrid retrieval enhancement based on SapBERT (Liu et al., 2021), a model specialized in biomedical entity linking.”