SciBERT: Pretrained Contextualized Embeddings for Scientific Text

By Iz Beltagy, Arman Cohan, Kyle Lo. 2019

228 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Release Strategies and the Social Impacts of Language Models (2019, background): “These include biomedical literature analysis [4], generating synthetic test data [29], and generating radiology reports [39] and EEG reports [7].”
Galactica: A Large Language Model for Science (2022, background): “At a higher level, we organize knowledge through natural language, and many works have trained on scientific text (Beltagy et al., 2019; Lewis et al., 2020a; Gu et al., 2020; Lo et al., 2019b).”
Multi-Stage Document Ranking with BERT (2019, neutral): “This result is in line with recent work that shows improvements with target corpus pre-training over out-of-domain corpus pre-training (Beltagy et al., 2019; Raffel et al., 2019).”
Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking (2019, methodology): “The current trend in research is to investigate all aspects of these language models, seeking insights in their inner workings (Tenney et al., 2019), or their application to various domains (Beltagy et al., 2019; Lee et al., 2019).”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “…biological properties [195], protein generation via language modelling [142], bioBERT [121] for text mining in biomedical data (with pretrained model and training code), embeddings of scientiﬁc text [18] (with code https: //github . com/allenai/scibert ) and medical question answering [237].”
Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs (2020, methodology): “we also compare against previous state-of-the-art (SOTA) models of BioBERT [17], SciBERT [6], and ClinicalBERT [14].”
Deduplication of Scholarly Documents using Locality Sensitive Hashing and Word Embeddings (2020, methodology): “, 2016), (Beltagy et al., 2019)) are also available apart from the one we used in this work.”
Entity, Relation, and Event Extraction with Contextualized Span Representations (2019, methodology): “We replaced BERT (Devlin et al., 2018) with SciB-ERT (Beltagy et al., 2019) which is pretrained on a large multi-domain corpus of scientiﬁc publications.”
Unsupervised Domain Adaptation of Contextualized Embeddings: A Case Study in Early Modern English (2019, methodology): “A similar approach is employed in SciBERT (Beltagy et al., 2019).”
CoTexT: Multi-task Learning with Code-Text Transformer (2021, methodology): “Similarly, SciBERT (Beltagy et al., 2019) is trained on the full text of biomedical and computer science papers.”
Bio-Megatron: Larger Biomedical Domain Language Model (2020, background): “Effectively transferring the success of BERT (De-vlin et al., 2018) to the biomedical domain, most notably Lee et al. (2019) (BioBERT) and Beltagy et al. (2019) (SciBERT) inspired a large number of similar works last year.”
Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow (2019, methodology): “14 million scientific papers (Beltagy et al., 2019).”
Hurtful words: quantifying biases in clinical contextual word embeddings (2020, methodology): “In Table 3, we report log probability bias scores assessed on the publicly-available SciBERT model [6], pretrained on biomedical text, and our baseline clinical BERT model, which further pretrains SciBERT on clinical notes.”
Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations (2019, background): “Other work extends BERT to domains where its original training data does not suffice (Beltagy et al., 2019; Lee et al., 2019).”
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (2019, methodology): “, 2019); SciBERT is a similar approach for scientific texts (Beltagy et al., 2019).”