ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing

By Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, G. Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas B. Fehér, Christoph Angerer, Martin Steinegger and 2 more. 2020

1,354 citations recorded by the source corpus.

Computational biology and bioinformatics provide vast data gold-mines from protein sequences, ideal for Language Models (LMs) taken from Natural Language Processing (NLP). These LMs reach for new prediction frontiers at low inference costs. Here, we trained two auto-regressive language models (Transformer-XL, XLNet) and two auto-encoder models (Bert, Albert) on data from UniRef and BFD containing up to 393 billion amino acids (words) from 2.1 billion protein sequences (22- and 112-times the entire English Wikipedia). The LMs were trained on the Summit supercomputer at Oak Ridge National Laboratory (ORNL), using 936 nodes (total 5616 GPUs) and one TPU Pod (V3-512 or V3-1024). We validated the advantage of up-scaling LMs to larger models supported by bigger data by predicting secondary structure (3-states: Q3=76-84, 8-states: Q8=65-73), sub-cellular localization for 10 cellular compartments (Q10=74) and whether a protein is membrane-bound or water-soluble (Q2=89). Dimensionality reduction revealed that the LM-embeddings from unlabeled data (only protein sequences) captured important biophysical properties governing protein shape. This implied learning some of the grammar of the language of life realized in protein sequences. The successful up-scaling of protein LMs through HPC to larger data sets slightly reduced the gap between models trained on evolutionary information and LMs.

Subfields: Artificial Intelligence, Developmental and Educational Psychology, Representation & Self-Supervised Learning, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Developmental and Educational Psychology

In Representation & Self-Supervised Learning

In Computational Biology & Bioinformatics

Citations in the corpus

Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[53] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Debsindhu Bhowmik, and Burkhard Rost.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “We also analyze 4 pre-trained Transformer models from ProtTrans (Elnaggar et al., 2020): ProtBert and ProtBert-BFD, which are 30-layer, 16-head BERT models; ProtAlbert, a 12-layer, 64-head ALBERT (Lan et al.”
Multi-Scale Representation Learning on Proteins (2022, methodology): “…sequence-based methods [Hou et al., 2018], methods partially pretrained on millions of sequences [Rao et al., 2019, Bepler and Berger, 2019, Elnaggar et al., 2020] as well as methods utilizing principles of geometric deep learning [Kipf and Welling, 2017, Derevyanko et al., 2018,…”
Transformer protein language models are unsupervised structure learners (2020, methodology): “We evaluate several pre-trained Transformer models, including ESM-1 (Rives et al., 2019), ProtBert-BFD (Elnaggar et al., 2020) and the TAPE Transformer (Rao et al., 2019).”
Language models enable zero-shot prediction of the effects of mutations on protein function (2021, methodology): “ESM-1v outperforms existing protein language models TAPE [22], UniRep [21], ProtBERT-BFD [14], and ESM-1b [12].”
SaProt: Protein Language Modeling with Structure-aware Vocabulary (2024, background): “Moreover, prominent PLMs like UniRep (Al-ley et al., 2019), ProtTrans (Elnaggar et al., 2021), ESM (Rives et al., 2019; Meier et al., 2021; Rao et al., 2021; Lin et al., 2022), and Evoformer (Hu et al., 2022; Jumper et al., 2021) have showcased outstanding performance across a diverse array of…”
Protein Representation Learning by Geometric Structure Pretraining (2022, methodology): “Results are evaluated with the scripts and predictions provided in (van Kempen et al., 2022).”
The language of proteins: NLP, machine learning & protein sequences (2021, methodology): “There have been works using contextualized embedding models (e.g. ELMO) on proteins for supervised-learning tasks (such as GO annotation, subcellular localization or structure prediction), as well as clustering sequences based on the learned representations [15,29,54,34].”
NetSurfP-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning (2022, background): “Recent breakthroughs in the field of natural language processing (NLP) (13) have allowed for generating information-rich representations of protein sequences without the explicit need for MSAs (14,15).”
A guide to machine learning for biologists (2021, methodology): “Transformer models have recently shown better accuracy than RNNs for tasks on biological sequences, but it remains to be seen whether these methods, which are often trained on billions of sequences using thousands of GPUs, will be able to outperform existing alignment-based methods of sequence analy”
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “In proteomics, protein language models are used for secondary structure and contact prediction (ProtTrans [127]), protein sequence generation (ProGen [128]), protein function prediction (ProtST [129]), major posttranslational modification prediction (ProteinBERT [130]), biophysical property…”
Protein tertiary structure prediction and refinement using deep learning and Rosetta in CASP14 (2021, positive): “12,23,24 We trained a version of the trRosetta network that uses TAPE language model embeddings 12 as additional input features (Figure 1 1(A)).”
ProtGPT2 is a deep unsupervised language model for protein design (2022, methodology): “Given both the growth of proteinsequence databases and the lack of annotation for a signi ﬁ cant part of the protein space, protein sequences have become great candidates for unsupervised training 4,10,11 and now offer the opportunity to encode and generate protein sequences.”