Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

By Amy X. Lu, Haoran Zhang, Marzyeh Ghassemi, Alan M. Moses. 2020

93 citations recorded by the source corpus.

Pretrained embedding representations of biological sequences which capture meaningful properties can alleviate many problems associated with supervised learning in biology. We apply the principle of mutual information maximization between local and global information as a self-supervised pretraining signal for protein embeddings. To do so, we divide protein sequences into fixed size fragments, and train an autoregressive model to distinguish between subsequent fragments from the same protein and fragments from random proteins. Our model, CPCProt, achieves comparable performance to state-of-the-art self-supervised models for protein sequence embeddings on various downstream tasks, but reduces the number of parameters down to 0.9% to 8.9% of benchmarked models. Further, we explore how downstream assessment protocols affect embedding evaluation, and the effect of contrastive learning hyperparameters on empirical performance. We hope that these results will inform the development of contrastive learning methods in protein biology and other modalities.

Subfields: Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Language models enable zero-shot prediction of the effects of mutations on protein function (2021, background): “1 Protein language models In the past few years, a number of groups have developed language models for protein sequences [23, 31, 14, 32, 33, 22, 13, 12].”
Self-supervised learning in medicine and healthcare (2022, methodology): “In addition to using generative techniques, a self-supervised contrastive-learning task can be constructed by creating positive protein fragments taken from the same protein sequence, as well as negative pairs from fragments from different sequences 43 .”
Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models (2020, methodology): “…short of the LSTM and ResNet models from Rao et al. (2019), but outperform both the Transformer model as well as all previous work proposing protein-specific pre-training tasks (Bepler and Berger, 2018; Lu et al., 2020), and a recent paper employing an auto-regressive LSTM (Alley et al., 2019).”
Protein Representation Learning by Geometric Structure Pretraining (2022, methodology): “…explored in existing methods include next amino acid prediction (Alley et al., 2019; Elnaggar et al., 2021), masked language modeling (MLM) (Rao et al., 2019; Elnaggar et al., 2021; Rives et al., 2021), pairwise MLM (He et al., 2021) and contrastive predictive coding (CPC) (Lu et al., 2020).”