The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics
By Hugo Dalla-torre, Liam Gonzalez, Javier Mendoza Revilla, Nicolás López Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Hassan Sirelkhatim, Guillaume Richard and 4 more. bioRxiv, 2024
552 citations recorded by the source corpus.
Closing the gap between measurable genetic information and observable traits is a longstanding challenge in genomics. Yet, the prediction of molecular phenotypes from DNA sequences alone remains limited and inaccurate, often driven by the scarcity of annotated data and the inability to transfer learning between prediction tasks. Here, we present an extensive study of foundation models pre-trained on DNA sequences, named the Nucleotide Transformer, ranging from 50M up to 2.5B parameters and integrating information from 3,202 diverse human genomes, as well as 850 genomes selected across diverse phyla, including both model and non-model organisms. These transformer models yield transferable, context-specific representations of nucleotide sequences, which allow for accurate molecular phenotype prediction even in low-data settings. We show that the developed models can be fine-tuned at low cost and despite low available data regime to solve a variety of genomics applications. Despite no supervision, the transformer models learned to focus attention on key genomic elements, including those that regulate gene expression, such as enhancers. Lastly, we demonstrate that utilizing model representations can improve the prioritization of functional genetic variants. The training and application of foundational models in genomics explored in this study provide a widely applicable stepping stone to bridge the gap of accurate molecular phenotype prediction from DNA sequence. Code and weights available on GitHub in Jax and HuggingFace in Pytorch. Example notebooks to apply these models to any downstream task are available on HuggingFace.
Subfields: Molecular Biology
View Paper: https://www.semanticscholar.org/paper/b06e1a2c84fb3bff03b10283bc863f007f5483b6
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Molecular Biology
- Stefano Ermon
- Christopher R'e
- Yoshua Bengio
- Stephen A. Baccus
- William L. Hamilton
- Yang Song
- Diederik P. Kingma
- Kyunghyun Cho
Citations in the corpus
- HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, methodology): “On fine-tuned benchmarks from the Nucleotide Transformer [10], HyenaDNA achieves state-of-the-art (SotA) on 12 of 18 datasets while using a model with orders of magnitude less parameters and pretraining data (see Tab.”
- Bilingual language model for protein sequence and structure (2024, unknown): “Thus, in computational biology Transformers have been limited to training large language models (LLMs) on amino (AA) and nucleic acid sequences [59].”