ProtGPT2 is a deep unsupervised language model for protein design

By Noelia Ferruz, Steffen Schmidt, Birte Höcker. 2022

866 citations recorded by the source corpus.

Protein design aims to build novel proteins customized for specific purposes, thereby holding the potential to tackle many environmental and biomedical problems. Recent progress in Transformer-based architectures has enabled the implementation of language models capable of generating text with human-like capabilities. Here, motivated by this success, we describe ProtGPT2, a language model trained on the protein space that generates de novo protein sequences following the principles of natural ones. The generated proteins display natural amino acid propensities, while disorder predictions indicate that 88% of ProtGPT2-generated proteins are globular, in line with natural sequences. Sensitive sequence searches in protein databases show that ProtGPT2 sequences are distantly related to natural ones, and similarity networks further demonstrate that ProtGPT2 is sampling unexplored regions of protein space. AlphaFold prediction of ProtGPT2-sequences yields well-folded non-idealized structures with embodiments and large loops and reveals topologies not captured in current structure databases. ProtGPT2 generates sequences in a matter of seconds and is freely available. Protein design aims to build novel proteins customized for specific purposes, thereby holding the potential to tackle many environmental and biomedical problems. Here the authors apply some of the latest advances in natural language processing, generative Transformers, to train ProtGPT2, a language model that explores unseen regions of the protein space while designing proteins with nature-like properties.

Subfields: Information Systems, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Molecular Biology

Citations in the corpus

Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models (2022, background): “Machine learning has also made an impact on sequence design both in the case of conditioning on structural information [3, 21], and without [17, 9, 37, 38, 16, 18, 31, 20].”
Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem (2022, positive): “Following the success of deep language models, Ferruz et al. (2022) developed protein sequence models to generate new proteins, but these models do not allow specification of structural motifs.”
De novo design of protein structure and function with RFdiffusion (2023, positive): “We explored two different strategies for training RFdiffusion: (1) in a manner akin to ‘canonical’ diffusion models, with predictions at each timestep independent of predictions at previous timesteps (as in previous work(5,8,9,16)), and (2) with self-conditioning(25), in which the model can conditio”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, neutral): “In contrast to protein sequences, which have had successes in protein language models [29, 31, 32, 17, 5, 41, 14], DNA sequences are orders of magnitudes longer (e.g. the human genome is 3.2B nucleotides) with long-range dependencies and interactions that span over 100k+ nucleotides in length [1].”
Protein structure generation via folding diffusion (2022, methodology): “Autoregressive training strategies have been successfully applied to generative modeling of sequences of language tokens in natural language processing 42,43 and amino acid tokens in protein language models 44 – 46 .”
Masked Inverse Folding with Sequence Transfer for Protein Representation Learning (2023, background): “Sequence-only protein language models A large body of work has recently studied the application of language models to sequence-only protein generation (Madani et al., 2020; 2021; Shin et al., 2021; Ferruz et al., 2022; Hesslow et al., 2022) and representation learning.”