MSA Transformer

By Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, Alexander Rives. 2021

735 citations recorded by the source corpus.

Unsupervised protein language models trained across millions of diverse sequences learn structure and function of proteins. Protein language models studied to date have been trained to perform inference from individual sequences. The longstanding approach in computational biology has been to make inferences from a family of evo lutionarily related sequences by fitting a model to each family independently. In this work we combine the two paradigms. We introduce a protein language model which takes as input a set of sequences in the form of a multiple sequence alignment. The model interleaves row and column attention across the input sequences and is trained with a variant of the masked language modeling objective across many protein families. The performance of the model surpasses current state-of-the-art unsupervised structure learning methods by a wide margin, with far greater parameter efficiency than prior state-of-the-art protein language models.

Subfields: Molecular Biology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Model Architectures

Citations in the corpus

Predicting multiple conformations via sequence clustering and AlphaFold2 (2023, positive): “To do this, we made predictions for DBSCAN-generated KaiB clusters in the model MSA Transformer 36 using its default settings.”
Highly accurate protein structure prediction with AlphaFold (2021, neutral): “The BERT objective is trained jointly with the normal PDB structure loss on the same training examples and is not pre-trained, in contrast to recent independent work 38 .”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, background): “(5) Will the combination of EI and AI [95] bring the best protein predictions of the future, or will the advantages of single-protein predictions (speed, precision) win out?”
ProGen2: Exploring the Boundaries of Protein Language Models (2022, background): “Protein language models have shown promise in representation learning for classiﬁcation, regression, and generation purposes (Rives et al., 2021; Rao et al., 2021, 2019; Madani et al., 2020; Meier et al., 2021; Elnaggar et al., 2020; 2 Related Work Large-scale language modeling: Language modeling…”
SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training (2021, positive): “Unlike the row attention used in [7, 17, 20, 33], intersample attention allows all features from different samples to communicate with each other.”
The language of proteins: NLP, machine learning & protein sequences (2021, background): “Even with deep models, the incorporation of expert knowledge and feature engineering can improve performance more than sophisticated models, especially with features that are not directly derived from the sequence, such as protein post-translational mod-iﬁcations or evolutionary information [80].”
SaProt: Protein Language Modeling with Structure-aware Vocabulary (2024, background): “Moreover, prominent PLMs like UniRep (Al-ley et al., 2019), ProtTrans (Elnaggar et al., 2021), ESM (Rives et al., 2019; Meier et al., 2021; Rao et al., 2021; Lin et al., 2022), and Evoformer (Hu et al., 2022; Jumper et al., 2021) have showcased outstanding performance across a diverse array of…”
Ultrafast end-to-end protein structure prediction enables high-throughput exploration of uncharacterized proteins (2022, positive): “In the MSA Transformer (16), the neural network model exploits the equivalence of aligned positions in the MSA in order to derive embeddings via a tied row attention framework.”
Exploring evolution-based & -free protein language models as protein function predictors (2022, background): “Like MSA-Transformer [27], Evoformer takes a family of evolutionarily-related and aligned protein sequences as input, in contrast to PLMs such as ESM-1b [32] and TAPE [25], which only take individual protein sequences.”
Toward the solution of the protein structure prediction problem (2021, methodology): “The single sequence model was recently extended to MSAs, outperforming current state-of-the-art methods such as trRosetta on contact prediction (191).”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Learning (2021, methodology): “We also thank the Leibniz Rechenzentrum (LRZ) for providing access to DGX-1(V100) for the testing phase.”