ProteinBERT: a universal deep-learning model of protein sequence and function

By Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, Michal Linial. bioRxiv, 2021

979 citations recorded by the source corpus.

Self-supervised deep language modeling has shown unprecedented success across natural language tasks, and has recently been repurposed to biological sequences. However, existing models and pretraining methods are designed and optimized for text analysis. We introduce ProteinBERT, a deep language model specifically designed for proteins. Our pretraining scheme consists of masked language modeling combined with a novel task of Gene Ontology (GO) annotation prediction. We introduce novel architectural elements that make the model highly efficient and flexible to very large sequence lengths. The architecture of ProteinBERT consists of both local and global representations, allowing end-to-end processing of these types of inputs and outputs. ProteinBERT obtains state-of-the-art performance on multiple benchmarks covering diverse protein properties (including protein structure, post translational modifications and biophysical attributes), despite using a far smaller model than competing deep-learning methods. Overall, ProteinBERT provides an efficient framework for rapidly training protein predictors, even with limited labeled data. Code and pretrained model weights are available at https://github.com/nadavbra/protein_bert.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/c07651110d3b98b63607557b57808d15d99013dd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “…(ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property prediction (PromptProtein [131]), and advancing the state of the art in proteomics [132,133].”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, background): “…protein language models have advanced the ability of machine learning models to predict protein function and fitness from sequence, especially when labeled training data is scarce Alley et al. (2019); Rives et al. (2021); Rao et al. (2019; 2020); Elnaggar et al. (2021); Brandes et al. (2021).”
PanGu-Coder: Program Synthesis with Function-Level Language Modeling (2022, background): “…e.g. to the Biomedical [37, 46, 9], Legal [14], Cyber Security [2], and Finance [6] domains, while simultaneously expanding to include signals from modalities other than natural language, e.g. Vision [65, 18, 15, 13, 64, 25], Proteins [11], Time Series [71, 72, 55] and Code [42, 70, 26, 31, 54].”
Genome-wide prediction of disease variants with a deep protein language model (2022, methodology): “A different deep-learning approach to functionalizing variants uses protein language models, a technique originally developed for natural language processing, which has been successfully applied to many protein prediction tasks [18–23].”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, neutral): “In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over 100k+ nucleotides in length [1].”