Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
By Alexander Rives, Siddharth Goyal, Joshua Meier, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, Rob Fergus. 2019
3,265 citations recorded by the source corpus.
Significance Learning biological properties from sequence data is a logical step toward generative and predictive artificial intelligence for biology. Here, we propose scaling a deep contextual language model with unsupervised learning to sequences spanning evolutionary diversity. We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect. Unsupervised representation learning enables state-of-the-art supervised prediction of mutational effect and secondary structure and improves state-of-the-art features for long-range contact prediction. In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and improving state-of-the-art features for long-range contact prediction.
Subfields: Molecular Biology, Representation & Self-Supervised Learning, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Molecular Biology
- Jinbo Xu
- Yang Zhang
- Kevin Kaichuang Yang
- David T. Jones
- Robin Pearce
- Sergey Ovchinnikov
- David Baker
- Burkhard Rost
In Representation & Self-Supervised Learning
- Yoshua Bengio
- Sebastian Ruder
- Caiming Xiong
- Richard Socher
- Yoav Goldberg
- Samuel R. Bowman
- Ruslan Salakhutdinov
- Trevor Darrell
In Scaling & Learning Theory
- Kevin Kaichuang Yang
- Stanislav Fort
- Maithra Raghu
- Samuel R. Bowman
- Dara Bahri
- Jian Peng
- Pushmeet Kohli
- Yi Tay
Citations in the corpus
- Evaluating Large Language Models Trained on Code (2021, background): “…et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal et al., 2020; Baevski et al., 2020), biology (Alley et al., 2019; Rives et al., 2021), and even across multiple modalities (Das et al., 2017; Lu et al., 2019; Ramesh et al., 2021; Zellers et al., 2021).”
- Rethinking Attention with Performers (2020, background): “…2018), document generation/summarization, time series prediction, generative modeling (e.g. image generation (Parmar et al., 2018)), music generation (Huang et al., 2019), and bioinformatics (Rives et al., 2019; Madani et al., 2020; Ingraham et al., 2019; Elnaggar et al., 2019; Du et al., 2020).”
- Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “Transformers (Vaswani et al., 2017) are ubiquitously state-of-the-art across many modalities, from language (Devlin et al., 2018; Raffel et al., 2019; Child et al., 2019) to images (Tan & Bansal, 2019; Lu et al., 2019) to protein sequences (Rives et al., 2019).”
- Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[14] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C.”
- High-resolution de novo structure prediction from primary sequence (2022, methodology): “…Helixon US Inc, USA ‡ Westlake Laboratory of Life Sciences and Biomedicine, Hangzhou, Zhejiang 310012, China § Computer Science & Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139 ¶ Correspondence 1 sufficient to predict the protein 3D structures ( 23 ).”
- Highly accurate protein structure prediction with AlphaFold (2021, positive): “In parallel, the success of attention-based networks for language processing 52 and, more recently, computer vision 31,53 has inspired the exploration of attention-based methods for interpreting protein sequences 54–56 .”
- Improved protein structure prediction by deep learning irrespective of co-evolution information (2020, methodology): “Recently, language models such as Transformer have been used to model protein sequences[33].”
- BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Sequence-only language models have since been trained through autoregressive or autoencoding self-supervision objectives for both discriminative and generative tasks, for example, using LSTMs or Transformer-based architectures [2, 9, 60, 64].”
- Generating functional protein variants with variational autoencoders (2020, background): “76 Deep sequence models such as LSTMs and transformers trained on datasets spanning the entire 77 range of known sequences have been shown to learn representations which distill structural and 78 functional information from the sequence [30, 31].”
- ProGen: Language Modeling for Protein Generation (2020, background): “Recent research (Alley et al., 2019; Rives et al., 2019; Rao et al., 2019) has begun to capitalize on the much larger set of raw protein sequences by adapting state-of-the-art representation learning techniques (Devlin et al., 2018) from natural language processing (NLP) to classification of protein…”
- Evolutionary context-integrated deep sequence modeling for protein engineering (2020, background): “It was found that using the learned representation as the feature input to fine-tune a supervised model improves fitness prediction on multiple protein mutagenesis datasets [28].”
- Modeling aspects of the language of life through transfer-learning protein sequences (2019, background): “Similar to the results described in [51], our projection of the embedding space confirmed that the model successfully captured bio-chemical and bio-physical properties on the most fine-grained level, i.e. the 20 standard amino acids (Fig.”
- Accelerating Protein Design Using Autoregressive Generative Models (2019, background): “Recently, autoregressive models have been applied across protein families to learn general features and predict mutation effects in unaligned protein sequences (Alley et al., 2019; Rives et al., 2019).”