SciBERT: A Pretrained Language Model for Scientific Text

By Iz Beltagy, Kyle Lo, Arman Cohan. 2019

3,836 citations recorded by the source corpus.

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et. al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.

Subfields: Artificial Intelligence, Molecular Biology, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Large Language Models

In Natural Language Processing

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “Targeted corpora can also produce domain-specific models (Beltagy et al., 2019), or broaden the expressiveness of models (e.g., for conversational modalities Adiwardana et al. (2020); Thop-pilan et al. (2022)).”
Llemma: An Open Language Model For Mathematics (2023, background): “Other work explores adapting pretrained models to novel domains by continued training (Rozière et al., 2023; Beltagy et al., 2019), parameter-efficient finetuning methods (Yong et al.”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “(Left) Example of a model page and model card for SciBERT (Beltagy et al., 2019), a pretrained model targeting extraction from scientific literature submitted by a community contributor.”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “[2020a] used this simple technique to create a “medical subset” of the MS MARCO passage retrieval test collection, which was then used to fine-tune a monoBERT model based on SciBERT [Beltagy et al., 2019].”
ExpBERT: Representation Engineering with Natural Language Explanations (2020, methodology): “To interpret explanations for Disease, we use SciBERT, a variant of BERT that is better suited for scientific text (Beltagy et al., 2019).”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “(Left) Example of a model page and model card for SciBERT (Beltagy et al., 2019), a pretrained model targeting extraction from scientific literature submitted by a community contributor.”
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks (2020, methodology): “For CHEMPROT and SCIERC, that is S2ORC-BERT (Lo et al., 2020), a similar model to SCIBERT.”
Pre-Trained Models: Past, Present and Future (2021, background): “Many works (Beltagy et al., 2019; Lee et al., 2020) further pre-train the general PTMs on this data to get better domain-specific or task-specific models.”
Continual Pre-training of Language Models for Math Problem Understanding with Syntax-Aware Memory Network (2022, background): “…most of previous works either continually optimize the model parameters with BERT-like tasks on domain or task related corpus ( e.g., scientific (Beltagy et al., 2019) and bio-media (Lee et al., 2020)), or design new pre-training objectives for task adaption ( e.g., commonsense reasoning (Zhou…”
Few-Shot Class-Incremental Learning for Named Entity Recognition (2022, methodology): “Following Beltagy et al. (2019); Souza et al. (2019), we use the BERT-CRF as our NER model, which consists of a BERT base (Devlin et al., 2018) encoder with a linear projection and a conditional random field (CRF) (Lafferty et al., 2001) layer for prediction.”
Explainable Automated Fact-Checking for Public Health Claims (2020, methodology): “S CI BERT is another BERT-based pre-trained model (Beltagy et al., 2019).”
S2ORC: The Semantic Scholar Open Research Corpus (2020, methodology): “We also release S2ORC-SCIBERT to serve as a baseline for research.”
Pre-trained models for natural language processing: A survey (2020, background): “Recently, some studies have proposed PTMs trained on specialty corpora, such as BioBERT [98] for biomedical text, SciBERT [11] for scientific text, ClinicalBERT [69, 3] for clinical text.”
Multi-Stage Document Ranking with BERT (2019, positive): “This result is in line with recent work that shows improvements with target corpus pre-training over out-of-domain corpus pre-training (Beltagy et al., 2019; Raffel et al., 2019).”
ExplainaBoard: An Explainable Leaderboard for NLP (2021, background): “This is true both for classical tasks such as machine translation (Sutskever et al., 2014; Wu et al., 2016), as well as for new tasks (Lu et al., 2016; Rajpurkar et al., 2016), domains (Beltagy et al., 2019), and languages (Conneau and Lample, 2019).”
Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents (2021, neutral): “…attempt to achieve performance gain by domain-adaptive pre-training on various domains, including the biomedical domain (Lee et al., 2020), the clinical domain (Alsentzer et al., 2019), the sci-entiﬁc domain (Beltagy et al., 2019), and the legal domain (Zhong et al., 2019; Chalkidis et al., 2020).”