Evolutionary-scale prediction of atomic level protein structure with a language model
By Zeming Lin, Halil Akin, Roshan Rao, Brian L. Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Y. Shmueli and 5 more. 2022
5,058 citations recorded by the source corpus.
Artificial intelligence has the potential to open insight into the structure of proteins at the scale of evolution. It has only recently been possible to extend protein structure prediction to two hundred million cataloged proteins. Characterizing the structures of the exponentially growing billions of protein sequences revealed by large scale gene sequencing experiments would necessitate a break-through in the speed of folding. Here we show that direct inference of structure from primary sequence using a large language model enables an order of magnitude speed-up in high resolution structure prediction. Leveraging the insight that language models learn evolutionary patterns across millions of sequences, we train models up to 15B parameters, the largest language model of proteins to date. As the language models are scaled they learn information that enables prediction of the three-dimensional structure of a protein at the resolution of individual atoms. This results in prediction that is up to 60x faster than state-of-the-art while maintaining resolution and accuracy. Building on this, we present the ESM Metage-nomic Atlas. This is the first large-scale structural characterization of metagenomic proteins, with more than 617 million structures. The atlas reveals more than 225 million high confidence predictions, including millions whose structures are novel in comparison with experimentally determined structures, giving an unprecedented view into the vast breadth and diversity of the structures of some of the least understood proteins on earth.
Subfields: Molecular Biology, Large Language Models, Computational Biology & Bioinformatics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Molecular Biology
- Michael Heinzinger
- Yang Zhang
- Burkhard Rost
- Martin Steinegger
- Johannes Söding
- Milot Mirdita
- David T. Jones
- Frank DiMaio
In Large Language Models
In Computational Biology & Bioinformatics
- Michael Heinzinger
- Burkhard Rost
- Martin Steinegger
- Yang Zhang
- Milot Mirdita
- Konstantin Weissenow
- Johannes Söding
- T. Ferrin
Citations in the corpus
- Generalized Biomolecular Modeling and Design with RoseTTAFold All-Atom (2023, positive): “Unlike previous protein-only deep-learning architectures ( 13 – 15 ), RFAA can model full bio-molecular systems.”
- AlphaFold2 and its applications in the fields of biology and medicine (2023, background): “AF2 as well as other machine learning algorithms (such as RoseTTAFold and recent language models 49,50 ) likely removes this obstacle.”
- End-to-end protein–ligand complex structure generation with diffusion-based generative models (2022, methodology): “Subsequent studies have proposed single-sequence structure prediction methods utilizing protein language model (PLM) representations [18–20].”
- Bilingual language model for protein sequence and structure (2024, positive): “…diverged sequences may adopt similar 3D structures [48], [49], we measured success in creating new sequences through the similarity in predicted 3D structure between the groundtruth (3D predicted by AFDB [30]) and predictions (3D structure predicted by ESMFold [6] for ProstT5-generated sequences).”
- Autonomous chemical research with large language models (2023, positive): “These models have been successfully applied to various domains, including natural language 1–5 , biological 6,7 and chemical research 8–10 as well as code generation 11,12 .”
- Accurate proteome-wide missense variant effect prediction with AlphaMissense (2023, background): “Although such models effectively capture the distribution of naturally evolved sequences, they lack the state-of-the-art understanding of protein structure achieved by AlphaFold (AF) ( 20 , 21 ).”
- Biasing AlphaFold2 to predict GPCRs and kinases with user-defined functional or structural properties (2022, neutral): “Since the emergence of AF2, a number of deep learning-based methods have been developed with the same goal of predicting protein structures at experimental accuracy (AlQuraishi, 2021; Baek et al., 2021; Chowdhury et al., 2022; Lin et al., 2022).”
- OpenFold: retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization (2023, neutral): “These include the ESMFold method for protein structure prediction 50 , which replaces MSAs with protein language models 51–53 , and FastFold, a community effort that has implemented substantial improvements including fast model-parallel training and inference 41 .”
- Automated model building and protein identification in cryo-EM maps (2024, positive): “The information in protein language models has been shown to be sufficient for protein structure prediction 14 .”
- SE(3) diffusion model with application to protein backbone generation (2023, methodology): “In this work, we use ProteinMPNN at temperature 0.1 to generate Nseq sequences for ESMFold (Lin et al., 2022) to predict structures.”
- Clustering predicted structures at the scale of the known protein universe (2023, background): “This difficulty is amplified by the fact that the repository of publicly available structures, consisting of the combined databases of AFDB and the ESMatlas 38 , is approaching a billion entries.”
- AlphaEvolve: A coding agent for scientific and algorithmic discovery (2025, neutral): “Other works use LLMs to optimize experimental planning and design [7, 10, 43, 75] or experiment execution and workflow [28, 62, 82, 105, 116].”
- UCSF ChimeraX: Tools for structure building and analysis (2023, methodology): “Using ESMFold to predict the individual domains of the insulin receptor produced accurate results with 1 – 3 Å C-alpha RMSD to the final cryo-EM structure (Figure 3D).”