Learning protein sequence embeddings using information from structure

By Tristan Bepler, Bonnie Berger. 2019

354 citations recorded by the source corpus.

Inferring the structural properties of a protein from its amino acid sequence is a challenging yet important problem in biology. Structures are not known for the vast majority of protein sequences, but structure is critical for understanding function. Existing approaches for detecting structural similarity between proteins from sequence are unable to recognize and exploit structural patterns when sequences have diverged too far, limiting our ability to transfer knowledge between structurally related proteins. We newly approach this problem through the lens of representation learning. We introduce a framework that maps any protein sequence to a sequence of vector embeddings — one per amino acid position — that encode structural information. We train bidirectional long short-term memory (LSTM) models on protein sequences with a two-part feedback mechanism that incorporates information from (i) global structural similarity between proteins and (ii) pairwise residue contact maps for individual proteins. To enable learning from structural similarity information, we define a novel similarity measure between arbitrary-length sequences of vector embeddings based on a soft symmetric alignment (SSA) between them. Our method is able to learn useful position-specific embeddings despite lacking direct observations of position-level correspondence between sequences. We show empirically that our multi-task framework outperforms other sequence-based methods and even a top-performing structure-based alignment method when predicting structural similarity, our goal. Finally, we demonstrate that our learned embeddings can be transferred to other protein sequence problems, improving the state-of-the-art in transmembrane domain prediction. 1

Subfields: Molecular Biology, Representation & Self-Supervised Learning, Computational Biology & Bioinformatics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Representation & Self-Supervised Learning

In Computational Biology & Bioinformatics

Citations in the corpus

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences (2019, neutral): “In contrast to models that are based on recurrent or convolutional neural networks, the Transformer makes no assumptions on the ordering of the input and instead uses position embeddings.”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, methodology): “Sequence-only language models have since been trained through autoregressive or autoencoding self-supervision objectives for both discriminative and generative tasks, for example, using LSTMs or Transformer-based architectures [2, 9, 60, 64].”
Evaluating Protein Transfer Learning with TAPE (2019, background): “73 Supervised [11] LSTM 0.”
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing (2020, background): “3%) compared to Transformer-based approaches? (2) Could redundancy- and noise-reduction of today’s largest data sets bring substantial changes? (3) Would the addition of auxiliary tasks such as next-sentence or sentence-order prediction offered by BERT or Albert suit protein sequences? A suggestion ”
Multi-Scale Representation Learning on Proteins (2022, methodology): “Bepler and Berger [2019] We downloaded the code and pretrained models from the ofﬁcial repository and embed the ligand sequence with a bidirectional LSTM.”
Energy-based models for atomic-resolution protein conformations (2020, background): “In parallel, recent work proposes that generative models pre-trained on protein sequences can transfer knowledge to downstream supervised tasks (Bepler & Berger, 2019; Alley et al., 2019; Yang et al., 2019; Rives et al., 2019).”
Evolutionary velocity with protein language models (2021, positive): “We quantify the sequence similarity as the Euclidean distance in language model-embedding space (Box 1), which can encode complex functional relationships (Bepler and Berger, 2019; Hie et al., 2021), and we construct the network by connecting a sequence to its k-nearest neighbors (KNN), which has be”
Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures (2021, methodology): “…by protein learning algorithms in different protein modeling tasks such as protein fold classiﬁcation (Hou et al., 2018; Rao et al., 2019; Bepler & Berger, 2019; Alley et al., 2019; Min et al., 2020) or protein function prediction (Strodthoff et al., 2020; Gligorijevic et al., 2019;…”
Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization (2020, methodology): “On protein-related downstream benchmarks [29], CPCProt achieves comparable results, despite using 0.9% the number of pretraining parameters of the largest model [3] and 8.9% of the number of parameters of the smallest neural network model [6], as illustrated in Figure 1.”
Machine-learning-guided directed evolution for protein engineering (2018, background): “However, more sophisticated continuous vector encodings of sequences can be learned from patterns in unlabeled sequences 52,70–76 or from structural information 77 .”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, methodology): “Representations of protein structure While we condition on structure and reconstruct sequence, there are other methods for incorporating protein structural information, such as predicting structure similarity between protein sequences [Bepler and Berger, 2019], corrupting and reconstructing the stru”
Protein sequence design with deep generative models (2021, background): “Additionally, while studies incorporating knowledge of protein structure are becoming increasingly powerful [9, 10, 11, 12, 13], they are beyond the scope of this review, and we focus on deep generative models of protein sequence.”
The language of proteins: NLP, machine learning & protein sequences (2021, methodology): “…models included a BERT-like Transformer pretrained on ~ 30 M Pfam domains, deep convolutional networks pretrained on 3D-structure contact prediction [13,116], LSTM language models [3], as well as non-deep-learning methods which excelled at some tasks (speciﬁcally secondary structure prediction).”