Transformer protein language models are unsupervised structure learners

By Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, Alexander Rives. 2020

394 citations recorded by the source corpus.

Unsupervised contact prediction is central to uncovering physical, structural, and functional constraints for protein structure determination and design. For decades, the predominant approach has been to infer evolutionary constraints from a set of related sequences. In the past year, protein language models have emerged as a potential alternative, but performance has fallen short of state-of-the-art approaches in bioinformatics. In this paper we demonstrate that Transformer attention maps learn contacts from the unsupervised language modeling objective. We find the highest capacity models that have been trained to date already outperform a state-of-the-art unsupervised contact prediction pipeline, suggesting these pipelines can be replaced with a single forward pass of an end-to-end model.1

Subfields: Molecular Biology, Representation & Self-Supervised Learning, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Representation & Self-Supervised Learning

In Computational Biology & Bioinformatics

Citations in the corpus

Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[34] Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives.”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, methodology): “Embeddings extracted from another large Transformer (ESMB-1b [67]), improved over all our non-ProtT5models (Fig.”
Language models enable zero-shot prediction of the effects of mutations on protein function (2021, background): “[24] found that transformer attention corresponds to known biological properties such as structure and binding sites and can be used to predict contacts.”
Guided Generative Protein Design using Regularized Transformers (2022, methodology): “Encouraged by the success of other works of Tamburini [2020] and Rao et al. [2020], we study the attention heads of our model for sequence-ﬁtness attribution.”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, neutral): “In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over 100k+ nucleotides in length [1].”
Exploring evolution-based & -free protein language models as protein function predictors (2022, methodology): “…these databases have been successfully applied for various protein related tasks, including secondary structure prediction [25], contact prediction [26], structure prediction [20], annotation prediction [32], signal peptide prediction [37], intracellular localization prediction [38],…”
SaProt: Protein Language Modeling with Structure-aware Vocabulary (2024, background): “Through self-supervised training on vast amounts of protein 1D residue sequences, PLMs have proven highly proficient in capturing long-range residue correlations, i.e., co-evolution (Anishchenko et al., 2017; Rao et al., 2020).”
Single-sequence protein structure prediction using language models from deep learning (2021, background): “We speculate that the latent space of the language model also captures recurrent evolutionary relationships 43 .”
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency (2022, methodology): “We also explored two other Transformer-based models: ESM trained on protein sequences (Rao et al., 2020) and Wav2Vec trained on audio data (Baevski et al., 2020).”
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies (2022, background): “Prior work has demonstrated that protein language models can learn structural features from sequence pre-training 55 alone (17, 33).”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, background): “…protein language models have advanced the ability of machine learning models to predict protein function and fitness from sequence, especially when labeled training data is scarce Alley et al. (2019); Rives et al. (2021); Rao et al. (2019; 2020); Elnaggar et al. (2021); Brandes et al. (2021).”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning (2021, methodology): “…or pre-training was measured. mmseqs2 was run on an Intel Skylake Gold 6248 processor with 40 threads, SSD and 377GB main memory, while protein LMs were run on a single Nvidia Quadro RTX 8000 with 48GB memory using half precision and dynamic batch size depending on sequence length (blue bar).”
Single-sequence protein structure prediction using language models and deep learning (2022, background): “We speculate that the latent space of the language model also captures recurrent evolutionary relationships 46 .”