BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR

By Souradip Chakraborty, Ekaba Bisong, Shweta Bhatt, Thomas Otto Friedrich Wagner, Riley Elliott, Francesco Mosconi. 2020

77 citations recorded by the source corpus.

The SARS-CoV-2 (COVID-19) pandemic spotlighted the importance of moving quickly with biomedical research. However, as the number of biomedical research papers continue to increase, the task of finding relevant articles to answer pressing questions has become significant. In this work, we propose a textual data mining tool that supports literature search to accelerate the work of researchers in the biomedical domain. We achieve this by building a neural-based deep contextual understanding model for Question-Answering (QA) and Information Retrieval (IR) tasks. We also leverage the new BREATHE dataset which is one of the largest available datasets of biomedical research literature, containing abstracts and full-text articles from ten different biomedical literature sources on which we pre-train our BioMedBERT model. Our work achieves state-of-the-art results on the QA fine-tuning task on BioASQ 5b, 6b and 7b datasets. In addition, we observe superior relevant results when BioMedBERT embeddings are used with Elasticsearch for the Information Retrieval task on the intelligently formulated BioASQ dataset. We believe our diverse dataset and our unique model architecture are what led us to achieve the state-of-the-art results for QA and IR tasks.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Verena Rieser
2. Gavin Abercrombie
3. Dirk Hovy
4. A. S. Bergman
5. Emily Dinan
6. Shannon L. Spruit
7. Y-Lan Boureau

In Natural Language Processing

1. Y-Lan Boureau
2. Emily Dinan
3. Gavin Abercrombie
4. Dirk Hovy
5. Shannon L. Spruit
6. Verena Rieser
7. A. S. Bergman
8. Jason Aaron Edward Weston

Citations in the corpus

• Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Medical advice Biomedical NLP is a large and active subfield, in which medicine-related automatic question answering is a widely studied task (see e.g. Chakraborty et al., 2020; Pergola et al., 2021).”
• Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation (2025, unknown): “The evaluated domain-pretrained models include SciBERT [35], PubMedBERT [40], Biomed-BERT [60], ChemicalBERT [42], ChemBERTa [43], and MatSciBERT [39].”