Modeling the language of life – Deep Learning Protein Sequences

By Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nachaev, Florian Matthes, Burkhard Rost. 2019

48 citations recorded by the source corpus.

Background One common task in Computational Biology is the prediction of aspects of protein function and structure from their amino acid sequence. For 26 years, most state-of-the-art approaches toward this end have been marrying machine learning and evolutionary information. The retrieval of related proteins from ever growing sequence databases is becoming so time-consuming that the analysis of entire proteomes becomes challenging. On top, evolutionary information is less powerful for small families, e.g. for proteins from the Dark Proteome. Results We introduce a novel way to represent protein sequences as continuous vectors (embeddings) by using the deep bi-directional model ELMo taken from natural language processing (NLP). The model has effectively captured the biophysical properties of protein sequences from unlabeled big data (UniRef50). After training, this knowledge is transferred to single protein sequences by predicting relevant sequence features. We refer to these new embeddings as SeqVec (Sequence-to-Vector) and demonstrate their effectiveness by training simple convolutional neural networks on existing data sets for two completely different prediction tasks. At the per-residue level, we significantly improved secondary structure (for NetSurfP-2.0 data set: Q3=79%±1, Q8=68%±1) and disorder predictions (MCC=0.59±0.03) over methods not using evolutionary information. At the per-protein level, we predicted subcellular localization in ten classes (for DeepLoc data set: Q10=68%±1) and distinguished membrane-bound from water-soluble proteins (Q2= 87%±1). All results built upon the embeddings gained from the new tool SeqVec neither explicitly nor implicitly using evolutionary information. Nevertheless, it improved over some methods using such information. Where the lightning-fast HHblits needed on average about two minutes to generate the evolutionary information for a target protein, SeqVec created the vector representation on average in 0.03 seconds. Conclusion We have shown that transfer learning can be used to capture biochemical or biophysical properties of protein sequences from large unlabeled sequence databases. The effectiveness of the proposed approach was showcased for different prediction tasks using only single protein sequences. SeqVec embeddings enable predictions that outperform even some methods using evolutionary information. Thus, they prove to condense the underlying principles of protein sequences. This might be the first step towards competitive predictions based only on single protein sequences. Availability SeqVec: https://github.com/mheinzinger/SeqVec Prediction server: https://embed.protein.properties

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[51] Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost.”
MSA Transformer (2021, background): “Alley et al. (2019) and Heinzinger et al. (2019) showed that LSTM language models capture some biological properties.”
Evaluating Protein Transfer Learning with TAPE (2019, background): “Recent work in protein representation learning has proposed a variety of methods that apply NLP-based techniques for transfer learning to biological sequences [11, 12, 30, 31].”
Transformer protein language models are unsupervised structure learners (2020, background): “…of large scale language models for natural language processing (Vaswani et al., 2017; Devlin et al., 2019), there has been considerable interest in developing similar models for proteins (Alley et al., 2019; Rives et al., 2019; Heinzinger et al., 2019; Rao et al., 2019; Elnaggar et al., 2020).”
Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization (2020, background): “…of these works directly take embedding techniques from natural language processing (NLP) tasks (Yang et al., 2018a; Riesselman et al., 2019; Rives et al., 2019; Alley et al., 2019; Heinzinger et al., 2019; Elnaggar et al., 2019; Armenteros et al., 2019; Madani et al., 2020; Elnag-gar et al., 2020).”
Single-sequence protein structure prediction using language models and deep learning (2022, background): “Very large transformer-based models trained on hundreds of millions and potentially billions of protein sequences are increasingly available 26,35,36 , and the scaling previously observed in natural language applications 47 makes it likely that the performance of RGN2 and similar methods will…”
Single-sequence protein structure prediction using language models from deep learning (2021, background): “Very large Transformer-based models trained on hundreds of millions and potentially billions of protein sequences are increasingly available 26,35,36 and the scaling previously observed in natural language applications 44 makes it likely that the performance of RGN2 and similar methods will…”