Language models enable zero-shot prediction of the effects of mutations on protein function

By Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, Alexander Rives. 2021

898 citations recorded by the source corpus.

Modeling the effect of sequence variation on function is a fundamental problem for understanding and designing proteins. Since evolution encodes information about function into patterns in protein sequences, unsupervised models of variant effects can be learned from sequence data. The approach to date has been to fit a model to a family of related sequences. The conventional setting is limited, since a new model must be trained for each prediction task. We show that using only zero-shot inference, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, performing at state-of-the-art.

Subfields: Molecular Biology, Large Language Models, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Large Language Models

In Computational Biology & Bioinformatics

Citations in the corpus

Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[54] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives.”
Learning inverse folding from millions of predicted structures (2022, methodology): “These likelihoods are inferred by masked language models, ESM-1b, ESM-1v, and ESM-MSA-1b, as described by Meier et al. (2021) (Table 3); additional details are given in Appendix C. Sequence insertions Using masked coordinate tokens at insertion regions, inverse folding models can also predict insert”
Protein Representation Learning by Geometric Structure Pretraining (2022, background): “, 2021), multiple sequence alignments (MSAs) (Rao et al., 2021; Biswas et al., 2021; Meier et al., 2021) and protein structures (Hermosilla et al.”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, background): “…2021; Chowdhury et al., 2021), encode evolutionary trajectories (Hie et al., 2022b; 2021), are zero-shot predictors of mutation fitness effects (Meier et al., 2021), improve out-of-domain generalization on protein engineering datasets (Dallago et al., 2021), and suggest improved sequences for…”
Exploring evolution-based & -free protein language models as protein function predictors (2022, methodology): “First, we observe that ESM-1b and ESM-1v [24] yield the best results, and both of them are evolution-free models.”
Genome-wide prediction of disease variants with a deep protein language model (2022, background): “1A ), was demonstrated to predict, without further training, variant effects correlated with the results of mutational scanning experiments [25].”
CADD v1.7: using protein language models, regulatory CNNs and other nucleotide-level scores to improve genome-wide variant predictions (2024, methodology): “Here, we include scores derived from the Evolutionary Scale Modeling (ESM) for assessment of variants in protein coding regions ( 33 ) as well as from a convolutional neural network (CNN) trained on open chromatin sequences, as a proxy for regulatory regions in the genome.”
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies (2022, methodology): “Embeddings from transformer encoder 28 models trained for masked language modeling have been used for variant prediction (19), evolutionary analysis (20, 21), 29 and as features for protein structure prediction (22, 23).”
Early computational detection of potential high-risk SARS-CoV-2 variants (2021, background): “Recently, Meier et al. demonstrated that these models also capture the effects of mutations on protein function [15].”
Efficient evolution of human antibodies from general protein language models (2023, positive): “Because we used general language models 19,20 , trained on non-redundant sequence datasets that are meant to represent variation across all natural proteins 23 , these models can only learn more general evolutionary rules than could a model trained specifically on antibody sequences 24–27 or a…”
ProGen2: Exploring the Boundaries of Protein Language Models (2022, methodology): “Protein language models have shown promise in representation learning for classiﬁcation, regression, and generation purposes (Rives et al., 2021; Rao et al., 2021, 2019; Madani et al., 2020; Meier et al., 2021; Elnaggar et al., 2020; 2 Related Work Large-scale language modeling: Language modeling…”
Single-sequence protein structure prediction using language models from deep learning (2021, methodology): “To enhance the capture of information in full protein sequences we introduce two training objectives not part of BERT or previously reported protein language models 26,32–36 .”
Scientific discovery in the age of artificial intelligence (2023, background): “…dominate natural language processing 37,99 and have been successfully applied to a range of problems, including earthquake signal detection 119 , DNA and protein sequence modelling 10,120 , modelling the effect of sequence variation on biological function 100,121 , and symbolic regression 122 .”