Bio-Megatron: Larger Biomedical Domain Language Model

By Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, M. Patwary, Mohammad Shoeybi, Raghav Mani. 2020

177 citations recorded by the source corpus.

There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books. Yet, most works do not study the factors affecting each domain language application deeply. Additionally, the study of model size on domain-specific models has been mostly missing. We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer. We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications. We demonstrate noticeable improvements over the previous state-of-the-art (SOTA) on standard biomedical NLP benchmarks of named entity recognition, relation extraction, and question answering. Model checkpoints and code are available at [this https URL] and [this https URL].

Subfields: Artificial Intelligence, Molecular Biology, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ajit Rajasekharan
2. Hoo-Chang Shin
3. Elvis Saravia
4. Brucek Khailany
5. Andrew Poulton
6. Sivanesan Sangeetha
7. Katikapalli Subramanyam Kalyan
8. Guillem Cucurull

In Molecular Biology

1. Ajit Rajasekharan
2. Sivanesan Sangeetha
3. Katikapalli Subramanyam Kalyan
4. Guillem Cucurull

In Large Language Models

1. Ajit Rajasekharan
2. Sivanesan Sangeetha
3. Katikapalli Subramanyam Kalyan
4. Thomas Scialom

In Natural Language Processing

1. Ajit Rajasekharan
2. Hoo-Chang Shin
3. Elvis Saravia
4. Sivanesan Sangeetha
5. Katikapalli Subramanyam Kalyan
6. Guillem Cucurull
7. Thomas Scialom

Citations in the corpus

• Galactica: A Large Language Model for Science (2022, background): “Scientific LanguageModels Works such as SciBERT, BioLM and others have shown the benefit of a curated, scientific corpus (Beltagy et al., 2019; Lewis et al., 2020a; Gu et al., 2020; Lo et al., 2019b; Gu et al., 2020; Shin et al., 2020; Hong et al., 2022).”
• A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, unknown): “BioBERT [91] FT PubMed, PMC Biomedical NER, RE, QA BlueBert [130] FT PubMed, MIMIC-III BLUE MIMIC-BERT [131] FT MIMIC-III Biomedical NER BioFLAIR [132] FT PubMed Bio NER Bio-ELECTRA-small [133] PT PubMed Biomedical NER AlphaBERT [134] FT Discharge diagnoses Extractive Summarization Task Spanish-bert”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “words using a single token but increases the size overall of the model [126].”
• VerilogEval: Evaluating Large Language Models for Verilog Code Generation (2023, background): “Question: Explain the high-level functionality of the Verilog module. assign {out[0],out[1],out[2],out[3],out[4], out[5],out[6],out[7] User Prompt: Question: Explain the high-level functionality of the Verilog module.”
• A large language model for electronic health records (2022, methodology): “Different from BioBERT and ClinicalBERT, the BioMegatron was trained from scratch without leveraging the original BERT model.”