BioBERT: a pre-trained biomedical language representation model for biomedical text mining
By Jinhyuk Lee, WonJin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, Jaewoo Kang. 2019
7,641 citations recorded by the source corpus.
Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.
Subfields: Artificial Intelligence, Health Informatics, Molecular Biology, Large Language Models, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Graham Neubig
- Noah A. Smith
- Yejin Choi
- Caiming Xiong
In Health Informatics
- Marzyeh Ghassemi
- Swaroop Mishra
- Stefan Harrer
- Matthew B. A. McDermott
- Maithra Raghu
- Jiawei Han
- Erica Schmidt
- Pranav Rajpurkar
In Molecular Biology
- Percy Liang
- Peer Bork
- Jure Leskovec
- Zhiyong Lu
- Lars Juhl Jensen
- C. von Mering
- Damian Szklarczyk
- M. V. Koroteev
In Large Language Models
- Noah A. Smith
- Zhiyuan Liu
- Jianfeng Gao
- Jie Tang
- Graham Neubig
- Percy Liang
- Jacob Eisenstein
- Weizhu Chen
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Noah A. Smith
- Kyunghyun Cho
Citations in the corpus
- SciBERT: A Pretrained Language Model for Scientific Text (2019, methodology): “Recent work on domain adaptation of BERT includes B IO B ERT (Lee et al., 2019) and C LINI - CAL B ERT (Alsentzer et al., 2019; Huang et al., 2019).”
- CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “In NLP, they were also shown to produce better language models, and are being widely used both for language representation and for classification in nearly every subfield (Tshitoyan et al. 2019; Gao et al. 2019; Lee et al. 2020; Feder et al. 2020).”
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, background): “…for domain adaptation Prior work has shown the benefit of continued pretraining in domain (Alsentzer et al., 2019; Chakrabarty et al., 2019; Lee et al., 2019).4 We have contributed further investigation of the effects of a shift between a large, diverse pretraining corpus and target domain…”
- Pre-Trained Models: Past, Present and Future (2021, background): “Some notable works include BioBERT (Lee et al., 2020) and SciBERT (Beltagy et al., 2019), which are trained respectively on the biological and scientific literature text.”
- Explainable Automated Fact-Checking for Public Health Claims (2020, background): “One example is BIOBERT, which is fine-tuned for the biomedical setting (Lee et al., 2020).”
- Pre-trained models for natural language processing: A survey (2020, methodology): “Recently, some studies have proposed PTMs trained on specialty corpora, such as BioBERT [98] for biomedical text, SciBERT [11] for scientific text, ClinicalBERT [69, 3] for clinical text.”
- PubMedQA: A Dataset for Biomedical Research Question Answering (2019, methodology): “We fine-tune BioBERT (Lee et al., 2019) on Pub-MedQA as a baseline.”
- Publicly Available Clinical BERT Embeddings (2019, background): “BioBERT (Lee et al., 2019) trains a BERT model over a corpus of biomedical research articles sourced from PubMed 3 article abstracts and PubMed Central 4 article full texts.”
- ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports (2022, methodology): “Efforts in this direction have been made, e.g., by Lee et al, Gu et al, and Singhal et al [20–22].”
- Continual Pre-training of Language Models (2023, background): “DAP-training can be achieved by directly updating the LM (Xu et al., 2019; Sun et al., 2019; Lee et al., 2020; Alsentzer et al., 2019; Gururangan et al., 2020; Chakrabarty et al., 2019; Ke et al., 2022b) or by training only a small set of additional parameters.”
- Federated pretraining and fine tuning of BERT using clinical notes from multiple silos (2020, neutral): “At the beginning of 2019, Lee et al. published a BERT model pre-trained on biomedical PubMed abstracts and PubMedcal central articles (Lee et al., 2019).”
- LEGAL-BERT: “Preparing the Muppets for Court’” (2020, background): “Being pre-trained on generic corpora (e.g., Wikipedia, Children’s Books, etc.) BERT has been reported to under-perform in specialised domains, such as biomedical or scientific text (Lee et al., 2019; Beltagy et al., 2019).”
- A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “)), most notably BioBERT (Lee et al., 2020) that was pre-trained on biomedical PubMED articles and SciBERT (Beltagy et al.”