Learning the Protein Language: Evolution, Structure and Function

By Tristan Bepler, Bonnie Berger. 2021

467 citations recorded by the source corpus.

Language models have recently emerged as a powerful machine learning approach for distilling information from massive protein sequence databases. From readily available sequence data alone, these models discover evolutionary, structural, and functional organization across protein space. Using language models, we can encode amino acid sequences into distributed vector representations that capture their structural and functional properties as well as evaluate the evolutionary fitness of sequence variants. We discuss recent advances in protein language modeling and their applications to downstream protein property prediction problems. We then consider how these models can be enriched with prior biological knowledge and introduce an approach for encoding protein structural knowledge into the learned representations. The knowledge distilled by these models allows us to improve downstream function prediction through transfer learning. Deep protein language models are revolutionizing protein biology. They suggest new ways to approach protein and therapeutic design. However, further developments are needed to encode strong biological priors into protein language models and to increase their accessibility to the broader community.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

State-specific protein–ligand complex structure prediction with a multiscale deep generative model (2022, positive): “Several works have demonstrated that deep generative models are capable of producing de novo-designed proteins with experimentally validated functions, including language models for protein sequence design 9,27,28 and diffusion models for protein backbone generation 29–32 .”
High-resolution de novo structure prediction from primary sequence (2022, methodology): “The OmegaFold model processes a protein sequence with a stack of GAU layers from ( 25 ) instaed of self-attention layers and multi-layer perceptrons.”
ProtGPT2 is a deep unsupervised language model for protein design (2022, methodology): “Supervised NLP methods, where the input sequences are trained jointly with their labels to produce predictive models, have been applied to various tasks, such as detecting structural similarity or predicting stability 3,4 .”
Protein Representation Learning by Geometric Structure Pretraining (2022, methodology): “Residue Type Prediction is based on the masked language modeling objective, which has been widely used in pretraining large protein language models (Bepler & Berger, 2021).”
Exploring evolution-based & -free protein language models as protein function predictors (2022, background): “Large-scale protein language models (PLMs) with self-supervised pretraining on tens of millions to billions of proteins (PLMs) [25, 32, 11, 4] are the curent state-of-the-art in predicting function and ﬁtness from sequences.”
Topsy-Turvy: integrating a global view into sequence-based PPI prediction (2022, methodology): “First, we generate a feature-rich representation of each protein using a protein language model (PLM) (Bepler and Berger, 2019, 2021); next, these features are combined using a convolutional neural network to predict interaction.”
FLIP: Benchmark tasks in fitness landscape inference for proteins (2021, background): “The vast majority of representation learning on protein sequences models entire sequences [27, 37, 15, 34].”
Dynamic-Backbone Protein-Ligand Structure Prediction with Multiscale Generative Diffusion Models (2022, methodology): “…NeuralPLexer only requires molecular graphs as ligand inputs, therefore can enable end-to-end gradient-based design for functional small-molecules and ligand-binding proteins when coupled to recently-proposed differentiable protein sequence [24–26] and molecular graph generators [27, 28].”
Efficient evolution of human antibodies from general protein language models (2023, unknown): “1c) to learn patterns that are likely to occur in natural proteins 15–22 .”