ProGen2: Exploring the Boundaries of Protein Language Models
By Erik Nijkamp, Jeffrey A. Ruffolo, Eli N. Weinstein, Nikhil Vijay Naik, Ali Madani. 2022
555 citations recorded by the source corpus.
Attention-based models trained on protein sequences have demonstrated incredible success at classification and generation tasks relevant for artificial-intelligence-driven protein design. However, we lack a sufficient understanding of how very large-scale models and data play a role in effective protein model development. We introduce a suite of protein language models, named ProGen2, that are scaled up to 6.4B parameters and trained on different sequence datasets drawn from over a billion proteins from genomic, metagenomic, and immune repertoire databases. ProGen2 models show state-of-the-art performance in capturing the distribution of observed evolutionary sequences, generating novel viable sequences, and predicting protein fitness without additional fine-tuning. As large model sizes and raw numbers of protein sequences continue to become more widely accessible, our results suggest that a growing emphasis needs to be placed on the data distribution provided to a protein sequence model. Our models and code are open sourced for widespread adoption in protein engineering. A record of this paper's Transparent Peer Review process is included in the supplemental information.
Subfields: Molecular Biology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Molecular Biology
- Alexander Rives
- Grant Goldman
- Charlotte H. Wang
- Vasilis Ntranos
- Chun Jimmie Ye
- Tom Sercu
- Brian L. Hie
- Robert Verkuil
Citations in the corpus
- Evolutionary-scale prediction of atomic level protein structure with a language model (2022, background): “[59] Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani.”
- Genome-wide prediction of disease variants with a deep protein language model (2022, background): “It has been shown that as language models scale in the number of parameters and training data, they tend to substantially improve [18, 46] (although this may not always be straightforward [47]).”
- Efficient evolution of human antibodies from general protein language models (2023, unknown): “1c) to learn patterns that are likely to occur in natural proteins 15–22 .”
- BloombergGPT: A Large Language Model for Finance (2023, background): “One popular domain is protein sequences since they can be represented using language-like sequences but are not covered by natural language models (e.g., Lin et al., 2022; Xiao et al., 2021; Nijkamp et al., 2022).”
- ProtGPT2 is a deep unsupervised language model for protein design (2022, methodology): “Protein autoregressive language models, such as ProGen 19 – 21 , RITA 22 , and DARK 23 have also been studied, and show the potential of autoregressive Transformers for protein design.”