GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics
By M. Zvyagin, Alexander Brace, Kyle Hippe, Yuntian Deng, Bin Zhang, Cindy Orozco Bohorquez, Austin R. Clyde, Bharati Kale, Danilo Perez-Rivera, Heng Ma and 24 more. 2022
142 citations recorded by the source corpus.
We seek to transform how new and emergent variants of pandemic-causing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pre-training on over 110 million prokaryotic gene sequences and fine-tuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole-genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.
Subfields: Infectious Diseases, Molecular Biology
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Infectious Diseases
In Molecular Biology
- Stefano Ermon
- Christopher R'e
- Yoshua Bengio
- Marcin J. Skwark
- Stephen A. Baccus
- Christian Dallago
- William L. Hamilton
- Yang Song
Citations in the corpus
- State-specific protein–ligand complex structure prediction with a multiscale deep generative model (2022, neutral): “…modelling include (1) auto-regressive models as widely adopted in transformer 24 networks for sequence data such as natural language and genomics 25,26 based on a sequential process and (2) diffusion-based generative models that leverage a stochastic process to generate data by sampling from a…”
- The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics (2024, background): “Large foundation models trained on sequences of nucleotides appear to be the natural choice to seize the opportunity in genomics research, with attempts to explore different angles recently proposed [18, 19, 19, 20].”
- HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, background): “…models (FMs) in genomics to learn generalizable features from unstructured whole genome data that can then be fine-tuned for a number of tasks including predicting the location and function of genes, identifying regulatory elements, and analyzing the evolution of species [25, 10, 20, 3, 53, 58].”