ProGen: Language Modeling for Protein Generation

By Ali Madani, Bryan McCann, Nikhil Vijay Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, Richard Socher. 2020

353 citations recorded by the source corpus.

Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ∼280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.

Subfields: Molecular Biology, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

In Large Language Models

Citations in the corpus

Rethinking Attention with Performers (2020, methodology): “Transformers (Vaswani et al., 2017; Dehghani et al., 2019) are powerful neural network architectures that have become SOTA in several areas of machine learning including natural language processing (NLP) (e.g. speech recognition (Luo et al., 2020)), neural machine translation (NMT) (Chen et al., 201”
Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models (2022, background): “Machine learning has also made an impact on sequence design both in the case of conditioning on structural information [3, 21], and without [17, 9, 37, 38, 16, 18, 31, 20].”
De novo protein design by deep network hallucination (2020, neutral): “in Ai = m 1, xp [ e − ( (F ) T i − F i−1 / )] (5)”
Generating functional protein variants with variational autoencoders (2020, background): “Other recent work has used deep autoregressive models without latent variables to handle raw sequence data, inspired by similar approaches in natural language processing [29, 30, 37].”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, methodology): “For TrEMBL, we used the exact same model parameters (nheads, nlayers, dff , d) = (8, 36, 1024, 512) from [34] for all runs.”
Luna: Linear Unified Nested Attention (2021, background): “…surprisingly versatile models that preform well on a wide range of language and vision tasks, including machine translation (Vaswani et al., 2017; Ott et al., 2018), language understanding (Devlin et al., 2019), image recognition (Dosovitskiy et al., 2020) and bioinformatics (Madani et al., 2020).”
Transformer protein language models are unsupervised structure learners (2020, background): “…processing (Vaswani et al., 2017; Devlin et al., 2019), there has been considerable interest in developing similar models for proteins (Alley et al., 2019; Rives et al., 2019; Heinzinger et al., 2019; Rao et al., 2019; Elnaggar et al., 2020; Lu et al., 2020; Madani et al., 2020; Shen et al., 2021).”
Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study (2023, background): “More generally, Transformer models have proven highly capable in generating music and images [20, 68], solving equations [39], performing logical and counterfactual reasoning with facts and rules defned using natural language [15], and generating proteins with desired properties [48].”
A Survey of Deep Learning for Scientific Discovery (2020, methodology): “…as training on protein sequences to ﬁnd representations encoding meaningful biological properties [195], protein generation via language modelling [142], bioBERT [121] for text mining in biomedical data (with pretrained model and training code), embeddings of scientiﬁc text [18] (with code https:…”
Bilingual language model for protein sequence and structure (2024, neutral): “Despite notable exceptions [7], [22], established pLMs are mostly limited to feature extraction from encoder-style Transformers [4]–[6].”