Protein language model embeddings for fast, accurate, alignment-free protein structure prediction

By Konstantin Weissenow, Michael Heinzinger, Burkhard Rost. 2021

124 citations recorded by the source corpus.

All state-of-the-art (SOTA) protein structure predictions rely on evolutionary information captured in multiple sequence alignments (MSAs), primarily on evolutionary couplings (co-evolution). Such information is not available for all proteins and is computationally expensive to generate. Prediction models based on Artificial Intelligence (AI) using only single sequences as input are easier and cheaper but perform so poorly that speed becomes irrelevant. Here, we described the first competitive AI solution exclusively inputting embeddings extracted from pre-trained protein Language Models (pLMs), namely from the transformer pLM ProtT5, from single sequences into a relatively shallow (few free parameters) convolutional neural network (CNN) trained on inter-residue distances, i.e. protein structure in 2D. The major advance originated from processing the attention heads learned by ProtT5. Although these models required at no point any MSA, they matched the performance of methods relying on co-evolution. Although not reaching the very top, our lean approach came close at substantially lower costs thereby speeding up development and each future prediction. By generating protein-specific rather than family-averaged predictions, these new solutions could distinguish between structural features differentiating members of the same family of proteins with similar structure predicted alike by all other top methods.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

1. Jianzhu Ma
2. Jian Peng
3. Jinbo Xu
4. Bonnie Berger
5. Shitong Luo
6. Sheng Wang
7. Shiwang Liu
8. Siqi Sun

Citations in the corpus

• High-resolution de novo structure prediction from primary sequence (2022, methodology): “Note that we omit the LayerNorm operation on the set variables such as { w ij } to simplify the notation.”
• Bilingual language model for protein sequence and structure (2024, unknown): “For general-purpose pLMs, this so-called transfer learning succeeds for many aspects of protein prediction, including, for function prediction: gene ontology [12], transport signals [13], binding residues [14], or subcellular location [15], [16], and for protein structure prediction: 2D [17] and 3D ”