Transfer learning enables predictions in network biology

By C. Theodoris, Ling Xiao, Anant Chopra, M. Chaffin, Z. A. Al Sayed, M. Hill, Helene Mantineo, Elizabeth M. Brydon, Zexian Zeng, X. S. Liu and 1 more. Nature, 2023

1,094 citations recorded by the source corpus.

Mapping gene networks requires large amounts of transcriptomic data to learn the connections between genes, which impedes discoveries in settings with limited data, including rare diseases and diseases affecting clinically inaccessible tissues. Recently, transfer learning has revolutionized fields such as natural language understanding^ 1 , 2 and computer vision^ 3 by leveraging deep learning models pretrained on large-scale general datasets that can then be fine-tuned towards a vast array of downstream tasks with limited task-specific data. Here, we developed a context-aware, attention-based deep learning model, Geneformer, pretrained on a large-scale corpus of about 30 million single-cell transcriptomes to enable context-specific predictions in settings with limited data in network biology. During pretraining, Geneformer gained a fundamental understanding of network dynamics, encoding network hierarchy in the attention weights of the model in a completely self-supervised manner. Fine-tuning towards a diverse panel of downstream tasks relevant to chromatin and network dynamics using limited task-specific data demonstrated that Geneformer consistently boosted predictive accuracy. Applied to disease modelling with limited patient data, Geneformer identified candidate therapeutic targets for cardiomyopathy. Overall, Geneformer represents a pretrained deep learning model from which fine-tuning towards a broad range of downstream applications can be pursued to accelerate discovery of key network regulators and candidate therapeutic targets. A context-aware, attention-based deep learning model pretrained on single-cell transcriptomes enables predictions in settings with limited data in network biology and could accelerate discovery of key network regulators and candidate therapeutic targets.

Subfields: Molecular Biology

View Paper: https://www.semanticscholar.org/paper/7d1e59ce254bea5228da634dbe7c5c4160df6f98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Molecular Biology

Citations in the corpus

Network pharmacology: towards the artificial intelligence-based precision traditional Chinese medicine (2023, methodology): “In this regard, AI methods, especially deep learning-based ones, have made great strides in automatically learning feature representations from biomedical data and thus are playing an increasingly important role in the field of TCM-NP [11].”
Towards Generalist Biomedical AI (2023, methodology): “[44] introduce Geneformer, a transformer [45] based model pretrained on a corpus of about 30 million single-cell transcriptomes to enable context-specific predictions in low data network biology applications.”
Empowering Biomedical Discovery with AI Agents (2024, positive): “These models have advanced modeling of proteins [22–26], genes [27], phenotypes [28], clinical outcomes [29–31], and chemical compounds [32, 33] through mining of biomedical data.”
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome (2023, positive): “…et al., 2022), chromatin state analysis (Lee et al., 2022), promoter-enhancer interaction prediction (Chen et al., 2022; Ni et al., 2022), TF-DNA binding prediction (Wang et al., 2022), variant effect prediction (Rozowsky et al., 2023), gene network prediction (Theodoris et al., 2023) and more.”
Large Scale Foundation Model on Single-cell Transcriptomics (2023, positive): “We then estimated the performance of various scale xTrimoGene architecture models with parameter sizes equivalent to previous transformer-based models 13–15 , and compared with scVI 25 (Supplementary Note 2).”